SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders
Authors: Zhuohao Yu
Xingru Jiang
Weizheng Gu
Yidong Wang
Qingsong Wen
Shikun Zhang
Wei Ye
Venue: NeurIPS 2025 poster ()
Keywords: Watermarking
Large Language Models
Sparse Autoencoders
Interpretability
Accountability of LLMs
AIGC Detection
Primary area: deep_learning
TL;DR: We embed personalized LLM Watermarks responsively into generated text with Sparse Autoencoders, with only inference-time sampling on black-box access LLMs and no extra training, and achieved high accuracy while preserving text quality.

Abstract: Watermarking LLM-generated text is critical for content attribution and misinformation prevention, yet existing methods compromise text quality and require white-box model access with logit manipulation or training, which exclude API-based models and multilingual scenarios. We propose SAEMark, an **inference-time framework** for *multi-bit* watermarking that embeds personalized information through *feature-based rejection sampling*, fundamentally different from logit-based or rewriting-based approaches: we **do not modify model outputs directly** and require only **black-box access**, while naturally supporting multi-bit message embedding and generalizing across diverse languages and domains. We instantiate the framework using *Sparse Autoencoders* as deterministic feature extractors and provide theoretical worst-case analysis relating watermark accuracy to computational budget. Experiments across 4 datasets demonstrate strong watermarking performance on English, Chinese, and code while preserving text quality. SAEMark establishes a new paradigm for **scalable, quality-preserving watermarks** that work seamlessly with closed-source LLMs across languages and domains.

SAEMARK: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders
Zhuohao Yu,, Xingru Jiang", Weizheng Gu, Yidong Wang, Qingsong Wen, Shikun Zhang, Wei Ye!
Peking University
zyu@stu. pku. edu. en, wyeēpku. edu. cn https://zhuohaoyu/ gi thub. iol..oo///aat io/SAEMark
Abstract
Watermarking LLM-generated text is critical for content attribution and misin- formation prevention, yet existing methods compromise text quality and require white-box model access with logit manipulation or training, which exclude API- based models and multilingual scenarios. We propose SAEMARK, an inference- time framework for multi-hit watermarking that embeds personalized information through feature-based rejection sampling, fundamentally different from logit-based or rewriting-barn based approaches: we do not modify model outputs directly and require only black-box access, while naturally supporting multi-bit message em- bedding and generalizing across diverse languages and domains. We instantiate the framework using Sparse Autoencoders as deterministic feature extractors and provide theoretical worst-case analysis relating watermark accuracy to computa- tional budget. Experiments across 4 datasets demonstrate strong watermarking performance on English, Chinese, and code while preserving text quality. SAE- watermarks
MARK establishes a new paradigm for scalable, that work seamlessly with closed-source LLMS across languages and domains.
Introduction
Watermark Generation
Watermark Detection
Targer Sequence
① Target Soy, Generation
Target Sequence Sentence
Targer Segurnce Jestence
Holing
Watemark
Geneate
Woneemark mrda
Geoerooe Torgen
244
Rejection Sampling
Thee sonteiiis
0852 X
2 Feat. Calculation
Peature Fatractor
Repbor Howkingene Jaee paaty
Frani
Pronpe
Eatractoe
X
sale caraling
155
fole the somn the
veuinpure
Ditarn
thil Earrsiii olat alows
diance look
liceradd cine spnd bus sude
clisiins
sccuici anl tn ficiro
eoees Gowoostttrr full issequell bz 2s.85090 eeem Narlinn Insehedd Usdes with and proprinety L.Me
own bodel
kedaneed pasio F1S10
proum diy moning to coverd
W
Feature Extractor: Sparse Auto-encoder
Impui
Spaase Autio crande
Fanure Concrmaaioon Scaee
Validation
remoine pare thevettial.
Geven St.
Super Emiclel
DAABl a.1) Grew
0.156
Wavemareed
tylor
Wiscataaked
Cao Be Reylaced By Any Dearmnniat ER shar slard Gountta Disrihuztss
Figure 1: An overview of SAEMARK.
:: Equal contribution. 1; Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Large language models (LLMs) have revolutionized text generation across domains, from creative writing to code synthesis [1. 2]. However, their ability produce human -ualltt text scale raises serious concerns about misinformation, copyright infringement, and content laundering. As these models become ubiquitous, reliably attributing Al-generated content becomes critical for accountability and trust.
Waternarking-embeedde detectable signatures into generated text-offers a promising solution. They must preserve text quality while enabling reliable detection, operate across languages and domains, and scale to distinguish between many users or sources. Most critically, they must work with real-world deployment constraints where model providers offer only API access without exposing internal parameters. The challenge becomes even more complex for multi-bit watermarking. Beyond simply detecting Al-generated text, the goal is to encode and recover specific message m E {0. - as user identifier for personalized attribution. This enables answering not just "is this Al-generated?" but "which specific user or system generated this text?" Such fine-grained attribution essential for large-scale deployment where accountability matters.
Existing watermarking methods struggle with these requirements. Token-level approaches like KGW [3] and EXP [4] require direct access to model logits, excluding APl-based deployment, and can degrade text quality through probability manipulation. Syntactic methods [5] fail to generalize across languages, while specialized approaches [6] work well in narrow domains but break down when applied more broadly. Even recent black-box methods [7, 8] rely on surface-level statisties or require auxiliary models, limiting their robustness and scalability.
We introduce SAEMARK, a fundamentally different approach that sidesteps these limitations entirely. Our key insight is deceptively simple: different LLM generations exhibit distinct patterns in their semantic features, and these patterns can be leveraged for watermarking through selection rather than modification. Instead of altering how text generated, we generate multiple candidates and choose those whose feature patterns align with a watermark key.
This approach works by operating on meaningful units of setteeennnees for natural language, functions for code. For each unit, we extract deterministic features that capture semantic properties, compute scalar statistic, and normalize it to behave predictably across different texts. Using the watermark key, we derive target values for each position. During generation, we sample multiple candidates from the LLM and select the one whose feature statistic is closest the target, ensuring the final sequence encodes the desired message. The elegance lies in what we don't change: no model weights, no logit manipulation, no token modifications. Every selected text segment is natural LLM output, preserving quality while enabling attribution. The approach works with any LLM through API calls, generalizes across languages and domains, and provides theoretical guarantees on watermark success that scale predictably with computational budget.
Our contributions span theory and practice. We develop a general framework for watermarking through reature-guided selection that works with any feature extractor and LLM. We provide theoret- ical guarantees that explain how SAEMark accuracy scales with compute budget, independent of feature extractors. Finally, we demonstrate a practical instantiation using SAE as feature extractor that achieves superior accuracy and text quality across languages and domains, encoding more information than existing approaches.
2 Related Work
LLM watermarking is a technique to embed special patterns into the output of LLMs, and has traditionally been used identify LLM generated text from human-written text [9]. Different
from post-hoc detection methods [10] that analyze statistical patterns in existing text, language
model watermarking aims to embed detectable signatures during generation [3]. These methods
compromise generation quality through direct manipulation of token probabilities [3] or syntactic
modifications [11]. The challenge of language and domain generalization remains largely unad- dressed, with current techniques primarily optimized for English and struggling with multilingual content or specialized domains like code [6]. Notably, PersonaMark [12] represents early attempts at personalized watermarking, but its reliance on English-specificc syntactic patterns and closed-source implementation makes scalability and cross- cross-lingual capability difficult to verify. Recently, more multi-bit watermarking methods have been proposed to embed multiple bits of information into gen-

---

erated text [13, 14, 15, 16, 17, 18), primarily by extending single-bit watermarking that manipulates logits during generation; these methods inherit the limitations of single-bit designs.
Complementary black-box watermarks avoid white-box logit access by using post-hoc selection or rewriting [8]. However, they typically operate statistics or introduce auxiliary model dependencies and do not directly address multi-bit message embedding at scale. Our framework differs by performing inference-time selection among naturally generated candidates using deterministic feature statistics. This enables extractor-agnoste analysis and multilingual, domain-agnostie multi-bit watermarking without modifying model logits.
Sparse Autoencoders (SAEs) are pre-trained interpretability tools that decompose LLM activations into unndeanndnndrsttdaalllleee features [19]. For given base model M and layer an SAE processes hidden states h, at position f. as:
f = SAE, (h:)
(I)
where f E IR' is a sparse vector (typically m > dim((h) with & 5% active features. The SAE is trained through two objectives: 1) reconstruct original activations, and 2) enforce feature sparsity via L1 regularization:
hb: - Dec(ff)) +
|felll
(2)
Lreg
Lxporec
This training produces features that correspond interpretable concepts [19, 20] like "Function definitions" or "Concept related to color blue" [21]. Our watermarking leverages key properties of pre-trained SAEs: multilingual activation allows the same features to fire for equivalent concepts across languages deterministically, sparsity enables efficient analysis through few active features per token. These properties support language-agnostic statistics via masked feature aggregations:
1
<(y)
Σ f
y
where m filters background features that fire ubiquitously regardless of content (e.g., punctuation). The summary o(y) provides deterministic statistic used by our watermarking procedures (Sec 3).
3 Methodology
We present general framework for post-hoc, multi-bit watermarking via feature-based rejection sampling. The key observation is that different LLM generations produce distinct values of determin- feature statistics computed over domain-approprin by selecting among naturally generated candidates, without modifying model logits, parameters or generated texts. We structure the section as follows: (1) general framework that extractor-agnoste (2) theoretical guarantees with an emphasis worst-case bounds, and (3) effective instantiation using sparse autoencoders as feature extractors.
units, and these statistics can be steered
3.1 Task Definition
We adopt multi-bit view of attribution: beyond binary detection, the objective is to encode a message m E {0, 1 that is recoverable at detection. Personalized attribution is a special case where m encodes user identifier bound to key.
Generation (multi-bit). Given base LLM M, watermark key k E K., input x, and message m E 13,, the algorithm produces by post-hoc selection over M's outputs (black-box/API- compatible; no access logits or parameters; ef. black-box watermarking [7, 8)):
y = Mark(M,, m).
Detection (multi-bit). For any text y and key k. the detection algorithm outputs a decoded message reject:
Detect(k, y') - m or T.
(5)
Threat model and scope: The scheme targets three properties: key privacy (deriving & from watermarked outputs is hard), verifier-held delectubility (any party holding can verify), and collusion resistance (multiple keys should not facilitate forgery). Our focus is attribution without storing LLM generated text. This work does not claim cryptographic unforgeability when keys are known; preventing adversarial forgeries is an important direction for security-focusee follow-ups.
3

---

3.2 General Framework for Feature-based Watermarking
Our approach operates on a simple intuition: suppose we have deterministic feature extractor that maps any text sequence into a scalar value, where such values follow predictable distribution (e.g.. approximately normal) for naturally generated text. Given a watermark key k encoding multi-bit information, we can derive a sequence of target scalar values from this key. During generation, we produce text chunk by chunk, ensuring each chunk yields a scalar value with the smallest difference to its corresponding effecte-ffectivelly implementing rejection sampling guided by reature-basee reward function. This process steers generation toward key-dependent patterns without modifying the underlying language model.
Algorithm 1: Watermark generation
Algorithm 2: Watermark detection
Input: Prompt key &. L.M extractor Ø,
Input:
Text I. candidate keys K., extractor Φ. statistic s(-) with CDF estimate F; alignment thresholds, significance Output: Detection result E K. (0)
statistic s(.) with CDF estimate F, units M,
attempts K, candidates N Output: Watermarked text x²
for attempt t- to K do
{zy} SegmentByDomain(2))
[F(s(((u)) Vu E
x² +- C. {T. M TarettFrrmmey(4): {za 0
D 0
for +- to do
foreach k E K. do
X' + GGnerateCamtiddeees(.......
{r, + TergetsFrommey(k: H2y
Whest - arg F(s(((z))))
if CheckAlignment((33
2, then
=EX
p + StudentTTestt [= (7))
x²' + x @ Tbesti + F(s()((ben)))
if > fa/2 Ap < a then D + DU ((k, end
end
if (heekkAligmme(((1) 1 then return
end end return (1, if of else
end end return
Figure 2: Pseudocode for SAEMARK: generation and detection.
Text segmentation. We segment text into smaller units {u; AM such as sentences for natural language or function blocks for code. Each unit will carry one symbol of the watermark signal.
Feature extraction. deterministic feature extractor Φ + Rdd maps each text unit to a feature vector, from which we compute scalar statistic s(u) ( E IR. Crucially, we assume that this statistic follows predictable distribution when computed over naturally generated text units.
Statistical normalization. To enable analysis independent of the specific feature extractor, normalize the statistic (()) to a standard range [0. 1] using its empirical distribution. Specifically, estimate the cumulative distribution function Fs from natural text, then map each unit's statistic via z()) F(s(w)) where f is the empirical CDF. This ensures =(t) values are approximately uniformly distributed for natural text.
Watermark generation process. Given a watermark key encoding multi-bit information, we first randomly generate sequence of target values {Ta Ad 1= by seeding PRNG generator with & and sampling each T; from a suitable range deterministically. Then, for each position i, we generate N candidate text units from the LLM and select the candidate c* whose normalized statistic 2(c") is closest to the target Tę-
Watermark detection process. To detect watermark input text, we segment into units, compute the normalized statistic zuu) for each unit, and compare the resulting sequence { against target sequences derived from candidate keys. We apply two-stage CheckAlignment process to verify sequence before statistical testing.
The CheckAlignment process employs two critical filters to ensure the observed sequence M M and expected target sequence [M are sufficiently similar:

---

Range Similarity Filter: This constraint ensures the dynamic ranges of observed and target sequences are similar:
Rmin
Fmax
Zmin
Rmax
(6)
Tmax
Tiin
where mmax max; 2is Amiu min; 2,, and similarly for We set Rmin = 0.95, Rmax 1.05.
Overlap Rate Filter: This constraint ensures sufficient overlap between the value ranges of both sequences:
{i 73 E (zmin Zmax
2 O min
M
where M denotes the number of textual units in the sequence and Omin 0.95 ensures that at least 95% of target values fall within the observed range. These two filters aim to eliminate spurious matches: the range similarity filter prevents matching sequences with fundamentally different statistical properties, while the overlap rate filter ensures meaningful correspondence between target and observed values. Only after passing both alignment checks do we apply Student's t-test for statistical significance. The key with the highest significance score is returned if it passes the threshold; otherwise, we classify the text as unwatermarked.
3.3 Theoretical Analysis and Guarantees
We provide theoretical guarantees on watermark embedding success that enable reliable detection by conservative bound. For clarity, we present our analysis for single textual unit and refer to our experiments for empirical validation of multi-unit performance with CheckAlignment process.
Embedding success under Gaussian assumption. Let target values т be sampled from the feasible range [μ 2o, f + 2a] where the feature statistic follows S ~ N NN,, o²). Given N candidate generations with feature statisties S2. S2..... ........ SN,, we seek the probability of finding least one candidate within relative tolerance of our target:
P(Jj 18, - kr)
Pmin) )
Worst-case analysis and bounds. To derive conservative guarantees, consider the worst-case target + 2a at the boundary of the feasible range. The single-candidate success probability becomes:
(1 (1+ (1 k) + 20) -H) (1 (11-))(u
k) + 20)
Pmin (P k)r S S₁ S (1+ = -
where & denotes the standard normal CDF and Pmin +2 (12(+k)
(22(1) -kppp/)))
The fundamental insight is that observed feature statistics are tightly bounded to target values. Setting strict tolerance k guarantees strong detection accuracy: embedding succeeds with high probability (1 Pmin) even with conservative parameters, while detection maintains precision because legitimate watermarks exhibit tight statistical binding that unwatermarked text cannot match. This framework provides exponential improvement with candidate count N, enabling principled compute- accuracy tradeoffs validated empirically across diverse tasks.
3.4 Sparse Autoencoder Instantiation
What concrete feature extractor should we use? We need statistics that are deterministic, seman- tically meaningful, and statistically regular. Sparse autoencoders auteencodess-ineppcabbiii tools designed to understand language model representations into interpretable semantic components (e.g. "technical writing." "math symbols") that exhibit distinctly different activation patterns across generations. By applying the SAE to separate "anchor" model, our approach remains compatible with any target language model, including API services, while extracting the discriminative yet predictable statistics our framework requires.
an ideal solution. They decompose language
The Feature Concentration Score intuition. Rather than using raw sparse autoencoder outputs, we compute a Feature Concentration Score (FCS) that captures a fundamental property of coherent text: semantic focus. The key insight is that well-formed text tends to concentrate its semantic activation
5

---

Probability Density
15
Sample Quantiles
0.25
KDE
Normal (u-0.142. 142,
0.20
0=0.029)
0.15
Normality Tests:
5
Kt Stat: 4.360 p-value: 0.113
0.10
KS Stat: 0.022
KS p-value: 0.730
0.05
0
0.0
0.1
0.2
0.3
2 Normal Theoretical Quantiles
-2
O
Feature Concentration Score
Figure 3: Distribution analysis of FCS. FCS distribution with density estimation (left) and Q-Q plot (right); statistical tests support approximate normality.
on a consistent set of relevant features, while unfocused or incoherent text spreads activation more uniformly. For example, technical manual concentrates activation features related formal language and domain expertise, while creative writing focuses on narrative and stylistic features.
This concentration pattern provides ideal watermark signal-we specific concentration levels without affecting text quality, since both high and low concentration can correspond to natural, well-written text in different contexts. The FCS measures this by identifying the most salient feature activated by each token, then computing what fraction of the total activation mass concentrates in these important features:
steer generation toward
FCS(T) = Ez= Eies Ut. Di=1lolll
(10)
contains the indices of the most salient features across all tokens, after applying the background mask m and deduplication. This provides our frame- work's statistic s(u) = Fw)( which empirically follows approximately normal distributions across different domains and languages, validating our theoretical assumptions. We provide illustration and detailed analysis of this process in Appendix D.
where farg (ox,
:t t
Implementation details. Two practical optimizations bridge the gap between our theoretical frame- work and robust empirical performance. First, the CheckAl ignment algorithm's eliminate spurious statistical matches that would otherwise compromise detection accuracy. Second, background feature masking ensures FCS calculations focus on discriminative semantic patterns rather than ubiquitous surface features. We precompute a mask excluding "background" SAE features, like those related to punctuation or basic grammar, to focus discriminative semantic patterns. With empirically observed parameters μ = 0.142, o = 0.029, and tolerance = 0.1, our bounds yield concrete success probabilities: N 50 achieves 99% per-unit success, N 20 maintains 85.32%, and N 10 achieves 61%. Since each generation involves multiple units, overall success rates significantly exceed these pei-unit bounds. Modern inference engines support parallel generation of N candidates simultaneously, making the approach practically efficient despite these extra compute overhead. We have extensive ablations and empirical results in the following experiments.
Experiments
Our experiments systematically address four fundamental questions: (1) How accurate and quality- preserving is our method compared existing single-bit and multi-bit watermarks? (2) What the computational overhead characteristics and scalability properties practice? (3) How robust is our method against adversarial attacks? (4) Which components contribute most significantly bridging the gap between theoretical bounds and empirical performance?
4.1 Experimental Setup

---

We evaluate on 4 diverse datasets as shown Table Following common practice in prior work, we report Accu- racy, Recall, F1 at 1% FPR. For text quality, we report win-rates of pair- wise comparison on PandaLM judged by GPT-4o in our main results, and average pointwise scores on BIGGen- Bench judged by their officially re- leased judge model as an alternative text-quality experiment. We use implementation for single bit
Table I: Dataset Statisties. Characteristics of the multilin- gual benchmarks used in evaluation.
C4 [22]
LCSTS 123]
MBPP [24]
PandaLM [25]
# Samples
500
500
257'
169
Language
English
Chinese
Python
English
Task Type
Completion
Summarization
Codo Generation
Instruction Following
test split of samitioed venion MBPP
watermarks from MarkLLM [26] toolkit and Waterfall [13] as it's the current best open-source training-free multi-bit watermark similar to our setting. Full details of baselines in Appendix C.
4.2 Watermarking Accuracy and Text Quality
Multi-bit watermarking poses fundamentally harder challenge than single-bit detection: must embed significantly more information into the same text length while maintaining both accuracy and quality. Despite this increased diffieulty, Table 2 shows SAEMark achieves superior accuracy compared to both single-bit baselines and the current best multi-bit watermark across all domains.
Accuracy across domains. SAEMark establishes new state-of-the-art performance: 99.7% F1 on English, 99.2% on Chinese, and 66.3% on code. Notably, outperform specialized methods in their own dlmmain--surpasiing code-specific SWEET by 3.9 points F1 (66.3% vs. 62.4%) despite our general-purpose design. While other methods suffer severe cross-domain performance cliffs, SAEMark captures language-agnostic patterns that generalize across syntactic variations. The multi- bit comparison reveals particularly dramatic advantages: SAEMark outperforms the current best multi-bit method Waterfall by 6.5 points F1 on English (99.7% vs. 93.2%) and an exceptional 54.7 points on code (66.3% vs. 11.6%), demonstrating semantic fature-based selection's clear superiority over vocabulary permutation approaches, especially in low-entropy domains.
Text quality. Beyond accuracy, Table shows SAEMark achieves the highest quality score (67.6%) on Mandal.M as judged by GPT-4o pairwise comparisons. To study how this generalizes across different backbone L.LMs, we conduct additional evaluation on BlGGen-Bench comparing against watermarked baselines and unwatermarked text.
Table 3 confirms SAEMark achieves the highest quality among watermarks across three backbone LLMs. This quality advantage stems from our design: rather than manipulating logits or applying external rewriting to obtain watermarked text, we simply run post-hoc selection among naturally generated candidates, ensuring text quality stays bounded by the its own capabilities.
Table 2: Comparison of Watermarks. We generate watermarked and unwatermarked texts and then report detection performance at 1% false positive rate (FPR), all in single-bit settings. Best results in bold and second-best are underlined. All metries are reported as percentages (%).
Method
C4 (English, (22]
LCSTS (Chinese, [23])
MBPP (Code, [24]
PandaL (Instruction, [25]
Ace.
Rec-t
Fif
Acc-t
Rec
FI?
Ace."
Rec.
H1
Quality:
Acc
Rec
HI
Single-bit Watermarks
KGW [3]
99.2
99.6
99.2
99.1
98.8
99.1
65.4
31.9
48.0
41.5
89.9
80.4
88.8
EXP [4]
99.5
99.6
99.5
99.3
99.4
99.3
57.8
16.7
28.4
23.2
79.3
59.4
74.2
UPV [27]
86.0
72.0
83.7
90.5
91.0
90.5
51.6
3.1
6.0
36.0
54.0
8.0
14.8
Unigram [28]
98.8
98.6
98.8
98.2
97.0
98.2
65.4
31.9
48.0
35.3
53.3
7.2
13.4
DIP [29]
96.0
92.6
95.9
97.7
96.2
97.7
60.7
22.6
36.5
36.5
81.5
63.8
77.5
Unbiased [30]
96.7
94.4
96.6
97.8
96.4
97.8
64.0
29.2
44.8
40.2
74.3
49.3
65.7
SynthID [31]
98.2
97.2
98.2
97.6
96.2
97.6
62.5
26.1
41.0
36.0
81.2
63.0
77.0
SWEET [6]
99.6
99.6
99.6
50.0
0.0
0.0
72.4
45.9
62.4
47.2
87.7
76.8
86.2
Multi-bit Watermarks
Waterfall [13]
93.6
88.0
93.2
95.3
91.6
95.1
52.5
6.2
11.6
46.4
73.2
47.1
63.7
SAEMARK (OURS)
99.7
99.8
99.7
99.2
99.6
99.2
74.5
50.2
66.3
67.6
86.6
73.9
84.6
7

---

Table Text quality evaluation on BIGGen-Bench. Scores are on a 5-point Likert scale (higher is better) [32].
Model
Unwatermarked SAEMark KGW Waterfall
Owenz. 1-n-tnst
4.13
4.05
3.97
4.02
Llama-3. 3. I-st-untt
3.69
3.85
3.56
3.62
gemma-3-4b-1
4.26
4.23
3.98
4.19
4.3 Computational Overhead and Scalability
Our theoretical analysis suggested requiring N=50 candidates achieve 99%+ accuracy per unit. However, through the CheckAlignmen filters, we achieve strong performance with significantly reduced computational overhead in practice.
two practical optimizations in our framework: background feature masking and
(a) Perf. vs. Sampled Candidates
(b) Perf. vs. End to end Latency
N=5
N=10
N=20
N=50
Method
Acc.
Rec.
F1
Latency
C4 [22]
Acc.
98.7
99.2
98.7
99.7
KGW
99.0
99.5
98.9
3.24x
Rec.
77.4
96.8
98.7
99.8
UPV
90.3
86.3
89.5
2.35x
F1
86.8
98.0
98.7
99.7
DIP
99.5
99.7
99.5
3,29x
LCSTS [23]
Acc.
98.6
99.0
98.6
99.2
Waterfall
98.8
97.3
98.1
1.06x
Rec.
72.6
96.0
98.0
99.6
F1
83.6
97.5
98.6
99.2
Ours(N=50)
99.5
99.7
99.5
1.00x
Figure 4: Computational overhead analysis. (a) Performance vs. number of sampled candidates for SAEMark. (b) Performance vs. avg latency across different watermarks.
Practical efficiency. Figure (a) demonstrates this efficiency gain: N=10 achieves 98.0% on English with reasonable overhead, while even N=5 attains 86.8% % s1-suustantially better than our conservative theoretical bounds predicted. This flexibility enables deployment across different computational budgets. Moreover, subfigure (b) reveals a remarkable result: SAEMark achieves 99.5% F1 at 1.00x baseline latency, substantially outperforming methods requiring 3.24x latency (KGW) and 3.29x latency (DIP) for comparable accuracy.
Infrastructure ad adnatagee This performance difference reflects a genuine architectural advantage. Since SAEMark requires no logit manipulation, we can leverage highly optimized inference backends like TGI with parallel candidate generation and tricks like prefix caching and custom, optimized CUDA kernels. In contrast, these optimized frameworks do not provide efficient watermark imple- mentations for logit-me manipulation methods, as such implementations require significant backend rewriting and may impact performance. While this creates a significant latency difference despite some methods theoretically needing less compute overhead, we consider this practical advantage reflecting the current state of inference infrastructure and these numbers reflect real deployment advantages of SAFMark.
Multi-bit scaling. Figure shows our approach maintains over 90% accuracy up to 10 bits (effectively differentiating 1,024 users) and 75% % accuracy 13 bits (8, 18,192 users), substantially exceeding Waterfall's performance through our high-dimensional SAE feature space. Importantly, method is only effective with 1,024 users- conducting fixed text length comparisons for fair evaluation. The superior information density stems from finer- iner-grained semantic distinctions our framework enables.
this does
4.4 Adversarial Robustness and Ablation Studies
Adversarial Robustness. Semantic SAE features provide inherent robustness against paraphrasing attacks. Figure demonstrates our method's resilience across three attack types-wond deletion, synonym substitution, and context-aware substitution. SAEMark shows strong resilience to such

---

Watermark Accuracy (%)
100
Accurra
90
True Positvve Rate
0.6
70
Ours (English)
Iettooe AUS 1558
Ours @ LCSTS (Chinese)
50
Waterfall (English)
40
Waterfall @ CCLCSS (Chinese)
02-
(Tdlyaaye per 201015 0955
64
128
256
512
1024
2048
4096
8192
0.2
Falise Positive Hate
0.8
Number of Users
Figure 5: Adversarial robustness. ROC curves showing robust perfor- mance against three attack types with varying intensities.
Figure 6: Multi-bit scaling and information density. Wa- termark across different message bits at fixed text length, demonstrating superior information density compared to multi-bit baselines with > 90% acc. up 10 bits.
attacks. Due space limitations, extended results testing attack intensities up 50% are provided in Appendix E, demonstrating continued robustness even under stronger attacks.
Ablation Studies primarily involves validating the following components" contribution to SAEMark's empirical success: CheckAlignmen filters. The Range Similarity and Overlap Rate filters prove the- oretically grounded and empirically validated. Figure 7 (left) demonstrates that the CheckAlignment algorithm's i 95% thresholds are not arbitrary-deviationn users (10 bits), confirming our theoretical analysis that these values optimally balance generation fea- sibility with discriminative power. These filters successfully compensate for theoretical independence assumptions when sequential generation creates dependencies in practice. Figure (right) shows that the empirically-dent parameters achieve optimal ROC performance, validating our framework theoretical foundations.
cause significant degradation beyond 1,024
Background feature masking. This implementation detail proves essential for signal quality. Figure 12 in the appendix shows that removing the background mask causes AUC to plummet from 1.0 0.85. The mask excludes ubiquitous features like those related to punctuation or basic grammar patterns that would otherwise dominate FCS calculations without providing discriminative signals between different watermark keys. Detailed ablation results are provided in Appendix E.
These components work synergistically to enable SAEMark's practical success: background masking isolates meaningful signals while alignment constraints makes watermark detection more accurate than the theoretical settings.
1.0
Accurayy (96)
100
Watermark Accuracy
True Positive Rate
0.6
80
SAUM (012 101
SAEMark
70
w/o Dynamic Range, Overlap
AUE OME
1233 201)
wjo Dynamic Range
Overlap
0.2
gie 1505
w/ Loose Umiits
50
0.0
2
612
1024
2048
4096
8192
0.0
0.2
0.4
0.6
0.8
1.0
Number Users
False Positive Rate
Figure 7: Framework component ablation studies. Left: Multi-bit watermarking accuracy scaling
analysis with ablations. Right: ROC curves for feature concentration hyperparams (μ, σ).

---

Conclusion
SAEMARK introduces a paradigm shift in Al-generated content attribution through feature- rejection sampling. Our approach addresses critical limitations of existing watermarking methods by operating entirely through inference-time selection rather than model modification, enabling deployment with API-based services while maintaining superior text quality and detection accuracy. Three key advances ensures success: First, our general framework provides theoretical guarantees that relate watermark accuracy computational budget, independent of the specific feature extractor used. Second, the sparse autoencoder captures meaningful semantic patterns that generalize across domains and languages and works great feature extractor. Third, practical optimizations bridge the gap between theoretical bounds and empirical performance, enabling efficient deployment. This work establishes that model interpretability tools can be effectively repurposed for content attribution tasks. The decoupling of watermarking from generation dynamics opens new possibilities for scalable, quality-presee preserving attribution systems that work seamlessly with existing language model APIs across diverse applications and languages.
10

---

References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[2] D. Guo, Q. Zhu, D. Yang. Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li et al., "Deepseek-coder: When the large language model meets programming- the rise of code intelli- gence," arXiv preprint vrXiv2441.14196 2024.
[3] Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, "A watermark for large language models," in International Conference on Machine Learning. PMLR, 2023, pp. 17 0771 084.
[4] S. Aaronson and H. Kirchner, "Watermarking gpt outputs." " 2022, https//wwww.sottaaronssoo cont/aalks/watermaak....
[5] A. B. Hou, J. Zhang, T. He, Y. Wang, Y.S.. Chuang, H. Wang, L. Shen, B. Van Durme, D. Khashabi, and Y. Tsvetkov, "Semsttmp: A semantic watermark with paraphrastic robustness for text generation," arXiv preprint arXiv:2310.03991, 2023.
[6] T. Lee, S. Hong, J. Ahn, I. Hong, H. Lee, S. Yun, J. Shin, and G. Kim, "Who wrote this code? watermarking for code generation," Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, 268/
pp. 4890-4911, 1. [Online]. Available:
[7] D. Bahri and J. Wieting, "A watermark for black-box language models," arXiv preprint arXiv:4410.0209,, 2024.
[8] Y. Chang, K. Krishna, A. Houmansadr, J. F. Wieting, and M. lyyer, "Postmark: A robust blackbox watermark for large language models," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 8969-8987.
[9] G. Jawahar, M. Abdul-Mageed, and Lakshmanan, "Automatic detection of machine generated text: A critical survey." arXiv preprint arXiv:201 2020.
[10] R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi, "Defending against neural fake news," Advances in neural information processing systems, vol. 32, 2019.
[11] M. J. Atallah, V. Raskin, C. F. Hempelmann, M. Karahan, R. Sion, U. Topkara, and K. E. Triezenberg, "Natural language watermarking and tamperproofing," in International workshop on information hiding. Springer, 2002, pp. 196-212.
[12] Y. Zhang, Lv, Liu, Y. Ma, Lu, X. Wang, X. Liu, and Liu, "Personamark: Personalized Ilm watermarking for model protection and user attribution," arXiv preprint arXiv:2409.09739, 2024.
[13] G. K. R. Lau, X. Niu, H. Dao, J. Chen, C.-S. Foo, and B. K. H. Low, "Waterfall: Scalable framework for robust text watermarking and provenance for Ilms," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 2042-220666
[14] R. Zhang, S. S. Hussain, P. Neekhara, and F. Koushanfar, " "(REMARK-LLM); A robust and efficient watermarking framework for generative large language models," " in 33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 1813-1830.
[15] K. Yoo, W. Ahn, J. Jang, and N. Kwak, "Robust multi-bit natural language watermarking through invariant features." arXiv preprint
2023.
[16] B. Guan, Y. Wan, Z. Bi, Z. Wang, H. Zhang, P. Zhou, and L. Sun, "Codeip: A grammar-guided bit watermark for large language models of code," arXiv preprint 15Xw,:2404.1563, 2024.
[17] K. Yoo, W. Ahn, and N. Kwak, "Advancing beyond identification: Multi-bi bit watermark for large language models." arXiv preprini 230.::300000022 2023.
[18] W. Qu, W. Zheng, Tao, D. Yin, Jiang, Tian, W. Zou, J. Jia, and J. Zhang, "Provably robust multi-bit watermarking for ai-generated text," arXiv preprint arXiv:2401. 16820, 2024.
11

---

[19]
T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Deni- son, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, Maxwell, N. Joseph, Z. Hatfield- Dodds, A. Tamkin, K. Nguyen, B. Mcl J. E. Burke, TT. Hume, S. Carter, TT. Henighan, and
C. Olah, "Towards monosemanticity: Decomposing language models with dictionary learn- ing." Transformer Circuits Thread, 2023, http::/traasformeecccrcuisssssp http:///////////oamanttic features/index.hmm.
[20] A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. McDougall, M. MacDiarmid, C. D. Freeman, T. R. Sumers, E. Rees, J. Batson, A. Jermyn, S. Carterr, C. Olah, and T. Henighan, "Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet." Transformer Circuits Thread, 2024. [Online]. Available: https:/trransfomme- www.uiisc.cciiiieeee 2ww.2/2/22eenninooooooooonniinid html
[21] T. Lieberum. S. Rajamanoharan, A. Commy, L. Smith, N. Sonnerat, V. Varma, J. Kramár, A. Dragan, R. Shah, and N. Nanda, "Gemma scope: Open sparse autoencoders everywhere all once on gemma 2," arXiv preprint arXiv:2408.08 05 147, 2024.
[22] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," Journal of machine learning research, vol. 21, no. 140, pp. 1-67, 2020.
[23] B. Hu, Q. Chen, and F. Zhu, "Lests: A large scale chinese short text summarization dataset," arXiv preprint arXiv: / 556605865. 2015.
[24] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q Le et al., "Program synthesis with large language models," arXiv preprint arXiv:2108.0m 07732, 2021.
[25] Y. Wang, Z. Yu, Z. Zeng, L. Yang, C. Wang, H. Chen, C. Jiang, R. Xie, J. Wang, X. Xie et al., "Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization," arXiv preprint
2023.
[26] L. Pan, A. Liu, Z. He, Z. Gao, X. Zhao, Y. Lu, B. Zhou, S. Liu, X. Hu, L. Wen el al. "Markllm: An open-source toolkit for IIm watermarking." arXiv preprint arXiv:2405.1005,, 2024.
[27] A. Liu, L. Pan, X. Hu, S. Li, L. Wen, I. King, and P. S. Yu, "A private watermark for large language models," arXiv preprint arXiv:2307.16230 2023.
[28] X. Zhao, P. Ananth, L. Li, and Y.-X. Wang, "Provable robust watermarking for ai- generaatttdd text," arXiv preprint arXiv:2306.1 17439, 2023.
[29] Y. Wu, Z. Hu, H. Zhang, and H. Huang, "Dipmark: A stealthy, efficient and resilient watermark for large language models," arXiv preprint arXiv:2310.07710 2023.
[30] Z. Hu, L. Chen, X. Wu, Y. Wu, H. Zhang, and H. Huang, "Unbiased watermark for large language models," arXiv preprint arXiv:2310. 10669, 2023.
[31] S. Dathathri, A. See, S. Ghaisas, P.-S. Huang, R. McAdam, J. Welbl, V. Bachani, A. Kaskasoli, R. Stanforth, T. Matejovicova et al., "Scalable watermarking for identifying large language model outputs," Nature, vol. 634, no. 8035, pp. 818-823, 2024.
[32] S. Kim, J. Suk, J. Y. Cho. S. Longpre, C. Kim, D. Yoon, G. Son, Y. Cho, S. Shafayat, J. Baek et al., "The biggen bench: A principled benchmark for fine- tineggrineed evaluation of language models with language models," arXiv preprint arXiv:2406.05700 2024.
[33] A. Nemecek, Y. Jiang, and E. Ayday, "Topic- based watermarks for lm-generated text," arXiv preprint arXiv:2404.002138 2024.
[34] L. Chen, Y. Bian, Y. Deng, S. Li, B. Wu, P. Zhao, and K.-f. Wong, "X-mark: Towards lossless watermarking through lexical redundancy," arXiv preprint 1rXXi:23311.088: 2023.
[35] V. S. Sadasivan, A. Kumar, S. Balasubramanian, W. Wang, and S. Feizi, "Can ai-generated text be reliably detected?"" arXiv preprini /1 2023.
[36] x. Zhao, L. Li, and Y.-X. Wang, "Dissillation-rssisaan watermarking for model protection in nip." " arXiv preprini arXiv:2210.03312. 2022.
[37] Z. Cai, S. Liu, H. Wang, H. Zhong, and X. Li, "Towards better statistical understanding of watermarking lms.. arXiv preprint arXiv:2403.13027, 2024.
12

---

[38] P. Molenda, A. Liusie, and M. J. Gales, "Waterjudge: Quality-detection trade-off when water- marking large language models," arXiv preprint Xii:24403.19548, 2024.
[39] X. Hu, P.-Y, Chen, and T.-Y. Ho, "Radar: Robust ai-text detection via adversarial learning." "
Advances in Neural Information Processing Systems, vol. 36, pp. 077-15 095, 2023.
[40] C. Gu, X. L. Li, P. Liang, and T. Hashimoto, "On the learnability of watermarks for language models," " arXiv preprint (rXiv:2312.0446) 2023.
[41] L. Chen, R. Qiu, S. Yuan, Z. Liu, T. Wei, H. Yoo, Z. Zeng, D. Yang, and H. Tong, "Wapiti: A watermark for finetuned open-source IIms," arXiv preprint 4rXiv:2410.06467, 2024.
[42] X. Zhao, C. Liao, Y.-X. Wang, and L. Li, "Efficiently identifying watermarked segments in mixed-source texts," arXiv preprint arXiv:2410.03660 2024.
[43] Q. Pang, S. Hu, W. Zheng, and V. Smith, "Attacking Ilm watermarks by exploiting their strengths," " in ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024.
[44] T. Gloaguen, N. Jovanović, R. Staab, and M. Vechev, "Black-box detection of language model watermarks," arXiv preprint MxXiv:2405.2000
2024.
[45] Z. Hu, L. Chen, X. Wu, Y. Wu, H. Zhang, and H. Huang, "Unbiased watermark for large language models," arXiv preprint arXiv:23 10. 10669, 2023.
[46] M. Mao, D. Wei, Z. Chen, X. Fang, and M. Chau, "A watermark for low-entropy and unbiased generation in large language models," arXiv preprint arXiv:2405. 2024.
[47] Y. Luo, K. Lin, and C. Gu, "Lost in overlap: Exploring watermark collision in llms," arXiv preprint arXiv:2403. 2024.
[48] M. Huo, S. A. Somayajula, Y. Liang, R. Zhang, F. Koushanfar, and P. Xie, "Token-specific watermarking with enhanced detectability and semantic coherence for large language models," arXiv preprint arXiv:2402.1 18059, 2024.
[49] L. Yang, S. Zhang. Z. Yu, G. Bao, Y. Wang. J. Wang, R. Xu, W. Ye, X. Xie, W. Chen et al., "Supervised knowledge makes large language models better in-context learners," arXiv preprint arXiv:2312. 15918, 2023.
[50] Z. Yu, C. Gao, W. Yao, Y. Wang, W. Ye, J. Wang. X. Xie, Y. Zhang, and S. Zhang, "Kieval: A knowledge groundedr interactive evaluation framework for large language models," arXiv preprint arXiv:2402. 15043, 2024.
[51] Z. Yu, C. Gao, W. Yao, Y. Wang, Z. Zeng, W. Ye, J. Wang, Y. Zhang, and S. Zhang, "Freeeval: A modular framework for trustworthy and efficient evaluation of large language models," " arXiv preprint arXiv:2404.0600 06003, 2024.
[52] R. Xie, Z. Zeng. Z. Yu, C. Gao, S. Zhang, and W. Ye, "Codeshell technical report," arXiv preprint arXiv:2403. 15747, 2024.
[53] Z. Yu, W. Gu, Y. Wang, Z. Zeng, J. Wang. W. Ye, and S. Zhang, "Outcome-refining process supervision for code generation," arXiv preprint arXiv:2412.15111
2024.
[54] W. Yao, Y. Wang. Z. Yu, R. Xie, S. Zhang, and W. Ye, "Pure: Aligning 11m via pluggable query reformulation for enhanced helpfulness." in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 8721-8744.
[55] Y. Wang, Z. Yu, Wang, Q. Heng, H. Chen, W. Ye, Xie, X. Xie, and S. Zhang, "Exploring vision-language models for imbalanced learning," International Journal of Computer Vision, vol. 132, no. 1. pp. 224-222- 2024.
[56] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., "Gpt-4o system card," arXiv preprint ar arX1v:2410.21276, 2024.
[57] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., "The Ilama 3 herd of models," arXiv preprint arXiv:2407.2178. 2024.
[58] OpenAI, "ol system card," https://cdnn.opeaiiccoo
aaasstem-crrdee Sep. 2024, accessed:
Dec 9, 2024.
[59] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al. "Palm 2 technical report," arXiv preprint 3rXiv:2305.10403. 2023.
13

---

[60] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin. C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems, vol. 35, pp. 27 230-27777700 744, 2022.
[61] OpenAI, "Gpt-4 technical report," 2023.
[62] Y. Bang, Cahyawijaya, N. Lee, W. Dai, D. Su, Wilie, H. Lovenia, Z. Ji. T. Yu, W. Chung et al., "A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity," arXiv preprint arXiv:2302. 04023, 2023.
[63] G. E. Hinton, S. Osindero, and Y. W. Teh, "A fast learning algorithm for deep belief nets," Neural Computation, vol. 18, pp. 1527-1554, 2006.
[64] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning. MIT Press, 2016,
vol. I.
[65] Y. Wang. Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, "Selffinstruct: Aligning language model with self generated instructions," arXiv preprint arXiv:2212.10566 2022.
[66] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. "Improving language understanding by generative pre-training," 2018.
[67] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith, "Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping." arXiv preprint arXiv:2002.063305 2020.
[68] C. Sun, X. Qiu, Y. Xu, and X. Huang. "How to fine-tune bert for text classification?" in Chinese Computational Linguistics: 18th China National Conference, CCL. 2019, Kunming, China, October 18-20, 2019, Proceedings 18. Springer, 2019, pp. 194-206.
[69] L. Hirschman and R. Gaizauskas, "Natural language question answering: the view from here," natural language engineering, vol. 7, no. 4, pp. 275-300, 2001.
[70] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee et al., "Natural questions: a benchmark for question answering research," Transactions of the Association for Computational Linguistics, vol. 7, pp. 453-466 466, 2019.
[71] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[72] L. Tunstall, L. Von Werra, and T. Wolf, Natural language processing with transformers. O'Reilly Media, Inc.", 2022.
[73] V. Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue, and Z. Nado, "Deep learning tuning playbook," 2023, version 1.0. [Online], Available: http://github.com// goom//oogle-reserchhuumm playeh//uunnnn__paooookk
[74] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and R. Bowman, "Glue: A multi-task bench- mark and analysis platform for natural language understanding." in International Conference on Learning Representations, 2018
[75] L. Yang, S. Zhang, L.
Qin, Y. Li, Y. Wang, H. Liu, J. Wang, X. Xie, and Y. Zhang, "Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective," arXiv preprint arXiv:2211.0l 1.00073. 2022.
[76] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love et al., "Gemma: Open models based on gemini research and technology." arXiv preprint arXhn:2403.08829, 2024.
[77] Z. He, W. Shu, X. Ge, L. Chen, J. Wang, Y. Zhou, F. Liu, Q. Guo, X. Huang, Z. Wu al., "Llama scope: Extracting millions of features from Ilama-3. --b with sparse autoencoders." arXiv preprint arXiv:2410.2055 2024.
[78] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodde R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah, "Toy models of superposition," Transformer Circuits Thread, 2022, https:// transformer- ciaooommerrcirrisssppiii cireuits.pub/2022/ee http://pmw.o//ooy.oooooooo.o.oo model/index.htmll
14

---

[79] R. Huben, H. Cunningham, L. Riggs, A. Ewart, and L. Sharkey, "Sparse autoencoders find highly interpretable features in language models," in The Twelfth International Conference on Learning Representations, ICIR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net,
2024. [Online]. Available: http:///eeeeeveeeee////rmmnm////
[80] A. Venugopal, J. Uszkoreit, D. Talbot, F. J. Och, and J. Ganitkevitch, "Watermarking the outputs of structured prediction with an application in statistical machine translation." in Proceedings of the 2011 Conference Empirical Methods in Natural Language Processing, 2011, pp. 1363-1372.
[81] R. Kuditipudi, J. Thickstun, T. Hashimoto, and Liang, "Robust distortion-free watermarks for language models," arXiv preprint arXiv:2307.1559; 2023.
[82] Y. Lu, A. Liu, D. Yu, J. Li, and I. King. "An entropy-based text watermarking detection method," arXiv preprint arXiv:2403. 13485, 2024.
[83] A. Yang, B. Yang. Zhang, B. Hui, Zheng. Yu, Ci Liu, F. Huang, H. Wei al., "Qwen2. 5 technical report," arXiv preprint arXiv:2412.15115, 2024.
[84] K. Krishna, Y. Song, M. Karpinska, J. Wieting, and M. lyyer, "Paraphrasing evades detectors of ai-generated text, but retrieval is effective defense," in Advances in Neural Information Processing Systems, A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. httpiiggsnnerrppp.c/pppeeeiii 575c45001 575c45000 3d0e99e4b0ecf82hdd afaa4- Paper-
27 469-27 500. [Online]. Available:
[85] A. Ng al., "Sparse autoencoder," CS294A Lecture notes, vol. 72, no. 2011, pp. 1-19, 2011.
[86] L. Wang, W. Yang, D. Chen, H. Zhou, Y. Lin, F. Meng, J. Zhou, and X. Sun, "Towards codable watermarking for injecting multi-bits information to llms," arXiv preprint arXiv:2307.15992. 2023.
[87] X. Xu, J. Jia, Y. Yao, Y. Liu, and H. Li, "Robust multi-bit text watermark with Ilm-based paraphrasers," arXiv preprint arXiv:2412.03122 2024.
15

---

A Limitations
We also found some limitations with our current approach. First, the method's effectiveness depends on SAE feature quality. But be noted that this does not affect the applicability of our algorithm on the base LLMs, since we only apply SAEs on the Anchor LLM and require only access to the output texts from the base LLM, and we have lot of pretrained SAEs from the open-source community that exhibit strong performance in interpreting model outputs. Second, detection watermarks effectively requires open-ended generation tasks, making attribution challenging for very short outputs like multiple-choice problems that only contain option keys. However this a universal challenge for all watermarking algorithms, since short texts inevitably contains less information and less space to inject additional signatures.
These constraints reflect tradeoffs in crivacy-preservinn watermarking. Future work could explore dynamic candidate pruning to address these limitations. Nevertheless, our experiments across benchmarks suggest these constraints pose manageable practical impacts compared to the system's s ethical advantages.
B
Experimental Setup and Hyperparameter Details
This appendix provides comprehensive description of the experimental setup, encompassing the hyperparameters and software configurations employed in this study.
B.1
Hyperparameters (SAEMARK)
The following hyperparameters used for the SAEMARK:
Candidate Number (N): 50. This parameter denotes the number of candidate sequences sampled from the LLM.
Unit Number (M): 10. This specifies the number of discrete generation units produced by the model per attempt.
Attempt Number (K): 15. This metric represents the maximum times that the algorithm attempts to get an alignment.
B.2 Model Configuration
The section outlines the hyperparameter by the model during generation.
Base Model: Qwen2.5-78- 5-78-Instruct. This is the model on which the algorithm operates.
Sampling: This algorithm enables the model to generate various candidates, for which the parameter do do_sample is set to True.
Temperature: This controls the randomness of the predictions by scaling the logits. The metric is set to 0.7.
Max New Tokens: This specifies the maximum number of new tokens that the model can generate, which is 20 during generation.
C Introduction to baselines
C.1
Single-bit Watermarks
KGW [3] The Key-based Green-list Watermarking (KGW) algorithm is modern approach for watermarking text generated by LLMs. This method builds upon the work [3], who introduced a watermarking scheme that divides the token set into 'red" and green' lists based secret key and previously generated tokens.
Key features of KGW include the bifurcation of the token set into 'red' and 'green' lists, the use of a random seed dependent on secret key and hash of prior tokens, reweighting of token log- probabilities to favor green tokens, and the introduction of permutation-basee reweight strategies.
16

---

These elements work in concert to create an effective watermarking system that balances detectability with output quality preservation.
The approach offers a balance between watermark embedding and preservation of text quality, addressing challenges faced by previous watermarking methods.
Unigram [28] The Unigram- Unigram-Watermatee Watermark and KGW algorithmss, both designed for watermarking LLM-generated text, have distinct characteristics. Unigram- Wattummakk operates on individual tokens, using a consistent green list for each new token, while KGW employs a K K-gram approach with varying green lists. Unigram- -atraam-rat 's simplicity offers enhanced robustness against editing attacks and requires minimal implementation overhead. This streamlined approach leads to potential efficiency gains in both watermark embedding and detection processes, setting apart from the more complex K- K-gram nature of KGW.
SWEET [6] The Segment- Wise Entropy- based Embedding Technique (SWEET) is an innovative approach to watermarking code generated by large language models. SWEET addresses the challenge of maintaining code functionality while embedding detectable watermarks. It operates by selectively applying watermarking high-entropy segments of the generated code, thereby preserving the overall code quality. This method significantly improves code quality preservation while outperforming baseline methods in detecting machine-generatee code. SWEET achieves this by removing low- entropy segments during both the generation and detection of watermarks, effectively balancing the trade-off between detection capability and code quality degradation.
UPV [27] The key feature of UPV is its use of separate neural networks for watermark generation and detection, addressing the limitation of shared key usage in previous methods. This separation allows for public verification without compromising the watermark's 's security. UPV employs shared token embedding parameters between the generation and detection networks, enabling efficient and accurate watermark detection. The algorithm embeds small watermark signals into the LLM's logits during generation, similar to existing methods, but uniquely conceals the watermarking details the detection process. This approach ensures high detection accuracy while maintaining computational efficiency, and significantly increases the complexity of forging the watermark, thus enhancing its security in public detection scenarios.
DIP [29] The Distribution- Preeetvvnnnn Watermarking (DIP) algorithm represents a significant advancement in watermarking techniques for large language models (LLMs). DIP's innovation is its ability to maintain the original token distribution of the LLM while embedding watermark, addressing critical limitation of previous methods. This cestribttion-prssrvvee property achieved through a novel permutation-basee approach that reweights token probabilities without altering the overall distribution. DIP offers provable guarantees on distribution preservation, detectability, and resilience against text modifications. The algorithm employs a texture key generation mechanism that considers multiple previous tokens, enhancing its robustness. Notably, DIP maintains text quality comparable to the original LLM output, owing to its distributionn nature.
Unbiased [30] Unbiased watermarking and DIP watermarking are closely related concepts in the field of text watermarking for large language models (LLMs). Both approaches aim embed watermarks while maintaining the original distribution of the LLM's output. The key distinction lies in their theoretical foundations and implementation. Unbiased watermarking ensures that the expectation of the watermarked distribution matches the original distribution, while DIP watermarking guarantees that the watermarked distribution is identical to the original for every input. In essence, unbiased watermarking can be viewed as a relaxed version of DIP watermarking. While unbiased watermarking allows for small deviations in individual instances, DIP watermarking maintains strict distribution preservation. This relationship highlights a spectrum of watermarking techniques, where unbiased methods offer balance between practicality and distribution preservation, while DIP methods provide stronger theoretical guarantees at potentially higher computational costs.
SynthID [31] SynthID is an advanced watermarking method for large language models (LLMs) that builds upon previous work in generative text watermarking. The key innovation of SynthID lies in its use of Tournament sampling, which provides superior detectability compared to existing methods. This approach offers rigorous and customizable non-distortion properties, allowing for
17

---

text quality preservation while maintaining effective watermarking, SynthID has been empirically validated, including through real user feedback from millions of chatbot interactions. Notably, the method introduces algorithm to combine generative watermarking with speculative sampling, enabling efficient deployment in high-performance, large-scale production L.LMs.
EXP [4] EXP employs pseudorandom function fa() with a secret seed s known only to the model provider. Given previous tokens Wi,..., w and GPT's probability distribution P1.....P PK for the next token Wi, the algorithm generates real values ri E [0, 1] using fs() EXP then selects the token i that maximizes !/Pi To detect the watermark, calculates Σ In i-r; In, and compares to a threshold. The scheme preserves the original token distribution while embedding detectable watermark, with theoretical analysis showing distinct expected values for normal and watermarked
1
text. The number of tokens required for reliable detection is O log }) where a is the average entropy per token and 5 is the acceptable misclassification probability.
C.2 Multi-bit Watermarks
C.2.1 Baselines
CTWL [86] CTWL is a framework designed to embed multi-bit customizable information into texts produced by large language models (LLMs). It allows watermarks carry details such model version, generation time, and user ID. CTWL provides a mathematical model for watermarking and comprehensive evaluation system that considers factors like success rate, robustness, coding rate, efficiency, and text quality. The Maruuinn method uses a proxy language model to partition vocabulary probabilities, aiming maintain watermarked text quality and achieve strong performance in evaluations. CT to integrate multi-bit information watermarks into LLMS and offers practical approach for tracing machine-generaten texts.
Waterfall [13] Waterfall a training-free framework designed for robust and scalable text wa- termarking. It leverages large language models (LLMs) paraphrasers to generate diverse text variations while preserving semantic meaning. By combining vocab permutation with orthogonal perturbation techniques, Waterfall aims achieve scalability and robust verifiability while main- taining text fidelity. The framework supports multi-bit watermarks, enabling it to accommodate multiple users while ensuring effective watermark detection. Waterfall allows for trade-off between watermark strength and text quality, making it adaptable to various requirements.
CODEIP [17] CODEIP embeds multi-bit message into generated code by softly biasing token logits during decoding. At each step, a hash of the previous token and the secret ID selects "watermark" tokens whose logits Lw are boosted proportional to strength | To guarantee syntactic validity, a pretrained type predictor assigns logits Lyp only to tokens matching the expected lexical category, scaled by y The final next-token logits combine the base LLM scores, watermark bias, and grammar bias via
Wi = arg max softmax(LLLM, + 3Lw + +yLpp)
weV
Extraction recovers the ID by re-hashing and finding the message maximizing cumulative watermark contributions.
REMARK-LLM [14] REMARK- introduces a robust watermarking framework for texts gen- erated by large language models. It consists of three modules: message encoding, reparameterization, and message decoding. The message encoding module uses a sequence-to- to-quueeee model to embed watermarks into --geerrrrte texts. The reparameterization module applies Gumbel-Softmal to the dense token distribution into a sparser form. The message decoding module extracts watermarks using a transformer-basee decoder. The framework incorporates malicious transformations during training to enhance robustness against attacks.
Provably Robust Multi-bit Watermarking for Al-generated Text [18] The authors propose
multi-bit watermarking scheme that embeds user's bit-string ID into L...--geeattedd text by first partitioning the message into rscudo-randomly assigned bit-segments per token, then biasing "green
18

---

Input: <Woukd>
<you>
<be>
<able>
<travel>
<through>
E.txta features
argmax/
Select the most significant feature
each token
2482
150
3. Agregate most significant fesatures
7.87
FCS
=0.1
||0,1|
61.9
Feature Set
SNYT
[=2
xeat
100
05
ai%
62
62
Feature Indicee
BX2
SAE Festuee Values
4. Calculate Feature Concentration Score
Figure 8: An example of Feature Concentration Score (FCS) calculation process.
list" logits seeded by each segment's s value. A dynamic- progrmmmmnggg step balances token-to teeen-tt-segment assignments, and Seed-Solomon error-correction layer encodes segments to correct editing errors. Extraction enumerates only each segment's ' possibilities O(k 25/k), yielding efficient, provably robust watermark recovery under bounded edit distance.
Robust Multi-hit Text Watermark with LLM-based Paraphrasers [87] The authors propose robust multi-bit text watermarking method using LLM-based paraphrasing. They design an encoder with two fine-tuned LLM paraphrasers (0p and 0,) that generate watermarked text by alternating based on the watermark code. A text segmentor divides the text into sentence-level segments, allowing each segment to encode one bit of the watermark message. The decoder uses trained text classifier to determine the watermark bit for each segment. The method employs a co-training framework where the encoder and decoder are alternately updated. The decoder acts as a reward model during PPO-based reinforcement learning to fine-tune the encoder, optimizing both the detection of the watermark and the semantic similarity the original text.
Robust Multi-bit Natural Language Watermarking through Invariant Features [15] The authors propose a robust multi-bir bit natural language watermarking method based on invariant features. They identify key words and syntactic dependencies as invariant features embed watermarks, leveraging these features' resistance to minor textual modifications. A corruption-resiste model is also introduced enhance watermark extraction robustness. Their method first selects mask positions based on these invariant features and then generates watermarked texts using an infill model. robust infill model developed to improve recovery of watermarked texts from corrupted versions.
infill
C.2.2 Why Choose Waterfall as the Multi-Bit Baseline?
Among the various text watermarking methods, Waterfall offers distinct advantages for multi-bit applications. As training-free framework, it efficiently generates diverse text variations using LLMs as paraphrasers while preserving semantic meaning. Its key advantages include complexity that doesn't 't depend word sentence count, allowing scalability. It also offers evaluation metrics akin to single-bit methods and supports multi-language and multi-datase watermarking, making it highly adaptable.
D Details of FCS Generation
This section elaborates on the methodology behind the generation of the Feature Concentration Score (FCS). The process is illustrated in Figure Figure 8, which outlines four key steps.
19

---

Algorithm 3: ComputeFCS(((T))
Input: Token sequence 7' SAE for the entire sequence
Output: Feature Concentration Score (FCS) <- ((T), yielding activation vectors Φ1, 92, On for cach token position in T;
indices < 0;
for =
do
Φ) is the activation vector token at position 4; inder <- arg max;(ch ( O m);;
Append index to indices;
end
I SeatureSet +- sel(indices), removing duplicates;
I JeatureSum um + 0; totalNorm +
for 1 to n do
token.Signsffcaance f- 0;
foreach E Sea do ficance tokenSignáftennce + Φi.;; tokenSigná Sicnniiinnaffaaeeeeee
end
featurr SeatureSum + featureSum + tokenSigni ficance:
LotalN orm <- lotalNorm + 1|o4|11:
// Accumulate significant features and norms
end
FCS Narm
SentareSum.
// Calculate final FCS
return FCS;
Extracting SAE Features for Each Token Given a token sequence T, we utilize SAE to derive an activation vector de for each token position This vector, Φr, embodies the representation of the token at position t with a dimensionality of 16,384.
Selecting the Most Significant Feature For every activation vector di, our objective is identify the most significant feature, which serves as descriptor for the token at position L. This achieved through applying the function argmazz (dh @ m); where is mask. The output of this function yields the indices corresponding to the most prominent feature, denoted as "SAE Feature Indices" in Figure Figure 8.
Aggregating Most Significant Features As depicted in Figure Figure each token's 's position / has its most significant feature. However, when summarizing the critical features of the entire sequence T, redundancies may occur. To address this, employ a operation to eliminate duplicate entries among the significant features, resulting in a unique collection termed as "Feature Set S""
Calculating Feature Concentration Score Upon obtaining the Feature Set S, we aim to quantify how these significant features contribute to the overall sequence concerning SAE feature values. For each de, we compute the sum of ór, where i represents the index belonging to S. This aggregate score measures the contribution significant features to individual tokens within Accumulating this metric across all tokens provides global measure for the sequence.
To evaluate the total activation value of SAE features over the sequence T. apply the L1 norm to each & obtaining the sum of absolute values for each token's 's feature vector. Summing these across all tokens yields the total SAE value for T. The Feature Concentration Score (FCS) is defined as the ratio of the accumulated contributions of significant features to the total SAE feature values.
The detailed steps for computing the FCS are outlined in algorithm 3.
This score effectively captures the concentration key features within a token sequence and is useful for applications in watermark embedding.
20

---

140
TTow
Jandee OLIRE
0.2
rtrydaue,, WHOLL AND BUSS
Amsp-torcceen, 2HO.3S ACC 6-30
AONPAAT AUCALAS
0.9
0.0
02
2.5
0.8
Falia Pustlive Raila
Figure 9: Word Deletion on SAEMark ROC curves highlighting the performance difference between "keep structure" and "not keep structure" methods under word deletion attacks with varying intensities (5%, 10%).
E Additional Experimental Results
E.1 Adversarial Robustness Evaluation
Word Deletion Attack In the main text, we conducted experiments using the "maintain content structure" version of the word deletion attack. However, the original word deletion attack involves splitting paragraph and randomly removing words, which disrupts the structure that watermarking methods rely on, making it harder for the detection system to identify the watermark. To address this issue, modified the attack to preserve structure while still performing word deletions. By maintaining the integrity of the structure, the attack bypasses watermark detection more effectively.
In our experimental results, we compare two versions of the word deletion attack. The "keep structure" method, represented
darker color, shows more robust performance with higher AUC values (0.949 at 0.05 and 0.858 at = 0.1). In contrast, the "not keep structure" method, shown in lighter color, demonstrates decline performance, with AUC values dropping to 0.901 at 0.05 and 0.825 at 0.1. These results indicate that preserving the content structure during the attack strengthens the watermark's resistance, whereas random word deletions that disrupt the structure reduce detection accuracy.
As shown in the Figure 9, the "keep structure" method outperforms the "not keep structure" method in terms of AUC, demonstrating its effectiveness in watermark resistance.
Basic Synonym Substitution Attack Our study also examines "keeping structure" versus "not keeping structure" approaches in the context of basic synonym substitution attacks, which are less likely to disrupt the content's structural integrity.
Figure 10 shows ROC curves comparing model performance under different conditions, with the original non-structuuee-presevviie
method in lighter shades and the modified uructure-preserving method darker hues. The analysis reveals minimal differences AUC values between the two, indicating similar model resilience to both forms of synonym substitution. Notably, the model demonstrates performance robustness that exceeds that observed in deletion attack scenarios, reflected by AUC scores that remain close the baseline.
Context-aware Synonym Substitution Attack Due to our algorithm's 's prominent performance against context-aware synonym attack. More intensities (20%, 30%, 40%, 50%) are carried upon these kinds of attacking.
The results of the context-aware watermarking method, shown in Figure 11 tested under this attack, demonstrate substantial robustness. Even with high substitution ratios-up to 50% token replacement- AUC remains relatively high, highlighting the method's 's ability maintain detection performance under significant adversarial pressure. The ROC curves further corroborate this, showing that the true positive rate remains consistently high across varying false positive levels, even attack
21

---

140
Thow Tatte
Jandee OLIRE
0.2
WHOLL AND 6AT)
kmip 2.0.35 ACC 6.3
Ionp-ET AUC.600
0.9
0.0
02
2.5
0.8
Falia Pusilive Raila
Figure 10: Basic Synonym Substitution on SAEMark ROC curves comparing "keep structure" and "not keep structure" methods under basic synonym substitution attacks different intensities (5%, 10%).
1.0
0.8
Thue Pootiiw
0.4
Tandes Sue
022
ADC-0.333
AUC.0821
2.5
Paloe Postiion Rist
Figure 11: Context-Aware Synonym Substitution on SAEMark ROC curves comparing "keep structure" and "not keep structure" methods under basic synonym substitution attacks at different intensities (5%, 10%).
intensity increases. This demonstrates well- ball-balanced trade-off between true and false positives, ensuring reliable detection without excessive false alarms. These findings affirm that the watermarking method is both effective and robust, offering reliable protection against sophisticated attacks while maintaining strong detection accuracy.
E.2 Ablation Study Background Frequent Features
In section 3, we utilize d 3 m, where m is a mask that excludes background frequent features. In this section, generate the Feature Concentration Score (FCS) without using and conduct ROC experiments for further analysis. To evaluate the impact of background frequent feature masking on model's performance, performed ablation study.
With background frequent feature masking in place, the model achieved an AUC of almost 1.0. Upon removing this masking, the AUC dropped to 0.85, as illustrated in Figure 12. This significant decrease demonstrates that background frequent feature masking plays a crucial role in our algorithm, emphasizing its importance for optimal performance.
22

---

140
0.2
Agroand Erigaeo: Instur minikng ADC1100 We bullgrouu Emqunnt bectire rasicins ACD 6.
0.9
0.0
02
2.5 Falon Pustive Raia
0.8
Figure 12: Ablation on Background Frequent Feature Masking The ROC curve compares the performance with and without background frequent feature masking.
F Use Of AI Assistants
We employed AI assistants for two tasks: (1) generating routine code implementations and boilerplate functions, and (2) performing grammatical review and sentence-level editing of the manuscript. All Al-generated content underwent thorough manual review. The core research methodology, findings, and analysis remain entirely
work.
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We support all claims with experiment results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a dedicated limitations section in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We provide proof in methodology section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report hyperparameters and we also open-source everything the linked mentioned in abstract.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We open-source everything in the linked mentioned in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c
public/gu Sttp://giidss/Coeeeeee wtw.mmmssssooooiiiiicooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. wttpccccppcccccccccccccccccccccc wwwgggiiiiiie weuuddes/ooooo des/CodeSubbi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We report all hyperparameters the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:In "methodology" section, t-test test method and report p-values in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the usage in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //neur///iiiee htpps.ccppublcctthccciuiieee
Answer:
[Yes]
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We report this in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks we do not train or release any model, and our datasets involved are all from existing, published, peer-reviewed works.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all related assets including models, data, code and baselines.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide does in our repository, including some one-line scripts to reproduce our experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
29

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We describe this in the appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy (https: (https://neurr) //tpe/rrps.cc/ooooeeeeeee/////// for what should or should not be described.
30


Reviews:
[{"review_id": "njc43wXrxN", "reviewer": "Reviewer_JSXp", "summary": "This paper proposes a novel, non-logit-manipulating LLM watermarking framework, SAEMARK, which embeds personalized signatures by leveraging the characteristics of Sparse Autoencoders (SAEs). This method addresses the limitations of existing watermarking schemes concerning text quality, white-box access restrictions, multilingual generalization, and user attribution. SAEMARK's core idea is to match feature distributions derived from a watermark key through rejection sampling after generation, thereby achieving watermarking without altering the original model's logits, while preserving text quality and watermark efficacy in multilingual and programming language contexts.", "questions": "The method uses rejection sampling to select texts that conform to a feature distribution, avoiding direct modification of logits and thus preserving text quality. However, this inherently requires the LLM to generate multiple candidate texts (e.g., N=50 as mentioned in the paper) until a suitable one is found.\n1.\tWhat is the exact computational overhead and inference latency introduced by this process of generating and filtering multiple candidates? Has the paper quantified this overhead compared to non-watermarked generation or other watermarking methods? Could this become a bottleneck in applications requiring high real-time performance?\n2.\tHow sensitive is SAEMARK's performance to the specific dataset used for training the Sparse Autoencoder? Are there best practices or guidelines for training robust SAEs for this specific watermarking purpose?\n3.\tHow to quickly distinguish between watermarked text and unwatermarked text? Is it necessary to match all users' keys and only after all failures can unwatermarked text be determined?\n4.\tThe paper lists 7 multi-bit watermarking methods, but only compares Waterfall. Is Waterfall superior to other methods on all datasets?\n5.\tThe public key is used for both generation and verification. The proposed method is how to prevent forgery while implementing public detection?", "limitations": "See weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tSince it does not rely on internal model logits or weight modifications, SAEMARK is compatible with API-based and closed-source LLMs, greatly expanding its applicability.\n2.\tOperating post-generation, by selecting texts that conform to the feature distribution via rejection sampling, it avoids modifying the model's logits or decoding algorithms, thereby maximizing the preservation of original text quality.\n\nWeaknesses\n1.\tWhile the paper emphasizes the advantage of rejection sampling in preserving text quality, the computational cost of repeatedly generating and filtering multiple candidate texts could be high.\n2.\tThe effectiveness of the method is influenced by the quality of features extracted by Sparse Autoencoders (SAEs). While pre-trained SAEs from the open-source community can be leveraged, the feature quality may still limit its performance in certain scenarios.\n3.\tAlthough it supports a large number of users, the accuracy gradually decreases as the user count grows exponentially.\n4.\tHigh repetition rates of rejection sampling may cause significant delays in the generation of watermark-containing text, which is often unacceptable to LLM service providers. The paper lacks relevant analysis.\n5.\tAs the number of users increases, the accuracy of watermark detection gradually decreases. Due to the increase in the number of keys, the complexity of repeating hypothesis testing also rises. The paper does not analyze the time and resource costs required to achieve multi-user detection.\n6.\tThere may be a risk of forgery when using the user's public key for sampling generation.\n7.\tAn unexpected line break is on Line 148.", "was_revised": true, "final_justification": "Thanks to the author for the careful rebuttal，I think that the borderline accept is appropriate for this paper.", "rebuttal": "Thank you for your thoughtful review and recognition of our method's innovation. We appreciate your detailed analysis and would like to address your concerns while clarifying the significant contributions of our work.\n\n**W1, 4 / Q1: Computational Cost & Applicability**\n\nYou raise valid concerns about the computational overhead of rejection sampling. To address this directly, we conducted experiments comparing performance under **controlled token generation budgets**—the actual bottleneck for both compute resources and latency. The results demonstrate that even with the same inference budget, SAEMark significantly outperforms state-of-the-art approaches:\n\nOur new results reveal that even **under the same inference token budget, our method significantly outperforms state-of-the-art approaches**. Here we use a typical bottlenecked scenario of long-text generation, where we allow generation of up to 10K tokens in English and Chinese.\n\n| **C4**        | Acc.     | Rec.     | F1       | Latency |\n| ------------- | -------- | -------- | -------- | ------- |\n| KGW           | 98.8     | 99.2     | 98.8     | 174s    |\n| UPV           | 86.1     | 74.2     | 84.2     | 94s     |\n| DIP           | 99.3     | **99.8** | 99.3     | 85s     |\n| Waterfall     | 98.7     | 97.0     | 97.9     | 52s     |\n| SAEMark(N=50) | **99.7** | **99.8** | **99.7** | **40s** |\n\n| LCSTS         | Acc      | Rec      | F1       | Latency |\n| ------------- | -------- | -------- | -------- | ------- |\n| KGW           | 99.2     | 99.8     | 99.0     | 85s     |\n| UPV           | 94.5     | 98.4     | 94.7     | 94s     |\n| DIP           | **99.6** | **99.6** | **99.6** | 178s    |\n| Waterfall     | 98.8     | 97.6     | 98.2     | **33s** |\n| SAEMark(N=50) | 99.2     | **99.6** | 99.2     | 40s     |\n\n\n\nAdditionally, although in our theoretical analysis we could achieve 99%+ accuracy with N=50 per unit, our empirical results yield high performance at N=10. **Using only 20% of compute overhead is sufficient to produce empirically strong performance**, this is because typical scenarios include multiple units instead of only one (typical LLM calls would generate multiple sentences at least). We now report single-bit watermarking performance with SAEMark given different sets of N (which is the most significant factor for compute cost).\n\n| C4             | Acc  | Rec  | F1   | AUROC |\n| -------------- | ---- | ---- | ---- | ----- |\n| SAEMark (N=5)  | 98.7 | 77.4 | 86.8 | 0.99  |\n| SAEMark (N=10) | 99.2 | 96.8 | 98.0 | 0.99  |\n| SAEMark (N=20) | 98.7 | 98.7 | 98.7 | 0.99  |\n| SAEMark (N=50) | 99.7 | 99.8 | 99.7 | 1.0   |\n\n| LCSTS          | Acc  | Rec  | F1   | AUROC |\n| -------------- | ---- | ---- | ---- | ----- |\n| SAEMark (N=5)  | 98.6 | 72.6 | 83.6 | 0.98  |\n| SAEMark (N=10) | 99.0 | 96.0 | 97.5 | 0.99  |\n| SAEMark (N=20) | 98.6 | 98.0 | 98.6 | 0.99  |\n| SAEMark (N=50) | 99.2 | 99.6 | 99.2 | 1.0   |\n\n\n\n**W2 / Q2: SAE Qualities**\n\nYour concern about SAE dependency touches on an important point, but we were perhaps too modest in our presentation. \n\nOur contribution is fundamentally **a paradigm shift in watermarking methodology**—we propose a general rejection sampling framework that is logit-free, training-free, and domain-agnostic. The use of SAE features is merely one instantiation—*you could replace Gemma Scope with any SAE, or even replace it with any deterministic feature that varies during generation*, *and our theoretical analysis in Sec 3.2 would **still hold*** as long as it follows a normal distribution. This flexibility opens up possibilities for more efficient implementations that could reduce the sampling overhead while maintaining the core benefits of our approach. Regarding SAE quality limitations, we want to highlight that our framework's modularity is actually a strength. As better, more robust SAEs become available from the open-source community, our method can immediately benefit without any algorithmic changes. The current results already demonstrate strong performance across multiple languages and domains, suggesting that existing SAE quality is sufficient for practical deployment.\n\n\n\n\n\n**W3: Users / Information Bits**\n\nYou correctly note accuracy degradation with exponential user growth.  This is indeed information-theoretically expected behavior, as the number of users is logarithmically related to the bits of information injected into watermarks. When you increase users from 1,024 to 8,192, you're essentially trying to fit 3 additional bits of information into the same text length. The graceful degradation (75% accuracy at 8,192 users) actually represents **state-of-the-art performance**. Waterfall, the best existing multi-bit method, can only reliably embed 8 bits with similar text length, while we achieve 10+ bits with higher accuracy.\n\n\n\n**Q3: Detection Efficiency**\n\nThank you for raising the point on quickly detecting watermarks.\n\nAs shown in Figure 3, SAEMark Detection requires a single FCS calculation pass (the bottleneck) followed by lightweight statistical tests. For large-scale deployment, we can use approximate nearest neighbor (ANN) search or KD-trees to quickly identify the K most likely user keys, then perform parallel statistical tests only on these candidates. This reduces detection from O(n) to O(log n) complexity for n users. This means detection would take milliseconds given millions of keys and FCS sequences.\n\n\n\n**Q4: Selection of baselines**\n\nYou ask why we only compared against Waterfall among multi-bit methods. We selected Waterfall as it represents the current state-of-the-art in multi-bit watermarking and has been shown to outperform other multi-bit approaches in recent benchmarks, and it is also open-source. Other multi-bit watermarks either require additional training (`Robust Multi-bit Text Watermark with LLM-based Paraphrasers` ), or provide no open-source implementation, or both.\n\nHowever, we agree that additional comparisons would strengthen our evaluation and can include results against other multi-bit methods in the revision.\n\n\n\n**Q5: Forgery Risks**\n\nYou raise an astute point about forgery risks. Our current focus is enabling user attribution without storing sensitive credentials, not preventing adversarial forgery. This is indeed an interesting direction for future security-focused research but can be slightly out of scope of this work.\n\n\n\nWe emphasize that SAEMARK achieves what no existing method can: **training-free, logit-free, multi-bit watermarking that works across natural languages and low-entropy domains like code**. This combination, along with supporting multi-bit watermarking with minimal quality degradation (we include additional quality-related experiments in our rebuttal to Reviewer R51M), addresses real deployment constraints in API-based LLM services. Even accounting for computational costs, the superior performance-per-inference and practical benefits make this a significant advancement for the field.\n\n\n\nWe hope these clarifications and additional experiments adequately address your concerns. Have we sufficiently demonstrated the practical viability and significant contributions of our approach? We remain open to further discussion and would be happy to provide additional clarifications or experiments that would help you better evaluate our work. Your feedback has already helped us strengthen the paper considerably, and we're committed to ensuring all your concerns are fully addressed."}, {"review_id": "EgzL21rx1V", "reviewer": "Reviewer_wvir", "summary": "This work introduces a novel technique for watermarking the outputs of black box LLMs based on extracting sparse codes from text chunks. They use techniques originally developed in the interpretability literature and combine a feature extractor (Anchor) LLM and a SAE model applied to its features to implement a multi-key(bit) watermarking scheme for attributing specific generations to specific keys (users). They empirically validate the performance of their method against existing LLM watermarking techniques and demonstrate an advantage in multi-key capacity, and detectability in domains like non-english text and code without impacting generation quality.", "questions": "1. Sec 3.1 Whats the JumpReLU \"jump\" hparam? and how is it tuned in this work?\n\n2. What are the meanings of k_priv and k_pub? It appears that kpub is used at both generation and detection time, i.e. this is not actually an asymmetric scheme. If the public key is used by both generation and detection then information sufficient to detect is information sufficient to embed, or spoof. It is fine if the method uses obscurity as part of the security process but using priv/pub notation is misleading to a skimming reader if the reviewer's understanding is correct.\n\n3. What is \"we employ Qwen-2.5-7B-Instruct [30] as the backbone model\" referring to? At L222 this is the first mention of \"backbone\". The only models expected to be necessary from the methodology section are the Anchor model the SAE and potentially an additional API model generating texts that will be rejection sampled since API-compatibility is discussed a lot in the introductory sections.", "limitations": "While the method and paper are implemented well (quality), the weaknesses discussed limit the significance of the method as the work is mostly an application of existing techniques to a new problem, and the also limit the originality in that other approaches address black box scenarios.\n\nAddressing some of the review concerns would improve the clarity aspects and potentially move the work in the direction of acceptance, whereas the primary weaknesses are mostly fundamental and can't be improved in the current submission. That said, situating the work more properly in the literature would also better the level of clarity in what the paper is contributing.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The novel application of an interpretability technique to the LLM watermarking problem.\n2. Compatibility of the approach with API models (black box) based on rejection sampling.\n3. Comprehensive evaluation of different watermarks, reasonable diversity of text domains/scenarios.\n\n\n### Weaknesses\n\n1. Clarity of the method. It is unclear how m in Eq 7, or \"Feature Set S\" in App. D, and Algo 3 is created. There is a mention of deduplication but the actual operation is unclear. This is a key part of the method without which the signal to noise would likely be poor and the method would underperform so it feels critical to describe precisely (reviewer's speculation because it is not ablated).\n\n2. Cost of the method. Technique requires N=50 to obtain 99% likelihood of _embedding_ the watermark (detecting it is separate), and this translates to 50x computational increase ignoring the prefill of the previously generated sequences/prompt, and assuming that sufficient diversity exists such that generations can/do not share a KV cache for the currently in progress textual unit. Batching can be used, and top-k response sampling is a standard practice in some generation frameworks on some hardware, but this issue is fundamental as serving batch capacity could always be used for other user requests etc. but this method requires more computation per individual request.\n\n3. Presentation of novelty, grounding to literature. Authors should note early in the Sec 3 Methodology that while the paper presents the SAE setup in an abstract form, it is not proposing this as a novel approach; Gemma Scope is simply being applied to this specific problem. A reader unfamiliar with Gemma Scope may think that this work is the first to propose the specific architectural setup for sparse feature extraction from the hidden states of modern LLMs. Also, this is not the first approach to study black box watermarking for arbitrary API-hosted LLMs via rejection sampling [1,2]. \n\n[1] \"A Watermark for Black-Box Language Models\" https://arxiv.org/abs/2410.02099\n\n[2] \"POSTMARK: A Robust Blackbox Watermark for Large Language Models\" https://aclanthology.org/2024.emnlp-main.506.pdf", "was_revised": true, "final_justification": "See responses.", "rebuttal": "Thank you for your insightful review and recognition of our novel application of interpretability techniques to watermarking. We deeply appreciate your thorough analysis and constructive feedback. Let us address your concerns comprehensively.\n\n**W1: Method Clarity**\n\nYou correctly identify areas needing clearer explanation. The mask **m** in Eq 7 excludes background frequent features (those occurring in >60% of samples), ensuring we focus on semantically meaningful variations rather than ubiquitous patterns. For Feature Set S creation, we apply Eq 7 to collect the most significant feature index for each token position, then perform deduplication to obtain unique features. This deduplication step is crucial—without it, frequently activated features would dominate the FCS calculation, reducing discriminative power between different watermark keys.\n\nWe agree that moving these implementation details from Appendix D to the main text would significantly improve clarity.\n\n**W2: Computational Cost & Practical Viability**\n\nYour computational overhead concerns are absolutely valid, and we've conducted extensive experiments addressing this directly. Our results reveal something surprising: **SAEMark achieves superior performance even under strictly controlled token generation budgets**. When comparing methods with equal computational resources for long-text generation (10K tokens):\n\n| **C4**        | Acc.     | Rec.     | F1       | Latency |\n| ------------- | -------- | -------- | -------- | ------- |\n| KGW           | 98.8     | 99.2     | 98.8     | 174s    |\n| UPV           | 86.1     | 74.2     | 84.2     | 94s     |\n| DIP           | 99.3     | **99.8** | 99.3     | 85s     |\n| Waterfall     | 98.7     | 97.0     | 97.9     | 52s     |\n| SAEMark(N=50) | **99.7** | **99.8** | **99.7** | **40s** |\n\n| LCSTS         | Acc      | Rec      | F1       | Latency |\n| ------------- | -------- | -------- | -------- | ------- |\n| KGW           | 99.2     | 99.8     | 99.0     | 85s     |\n| UPV           | 94.5     | 98.4     | 94.7     | 94s     |\n| DIP           | **99.6** | **99.6** | **99.6** | 178s    |\n| Waterfall     | 98.8     | 97.6     | 98.2     | **33s** |\n| SAEMark(N=50) | 99.2     | **99.6** | 99.2     | 40s     |\n\nRemarkably, SAEMark achieves the **lowest latency on English** while maintaining highest accuracy. Furthermore, while our theoretical analysis suggests N=50 for 99%+ success, empirical results show strong performance with much smaller N:\n\n| C4             | Acc  | Rec  | F1   | AUROC |\n| -------------- | ---- | ---- | ---- | ----- |\n| SAEMark (N=5)  | 98.7 | 77.4 | 86.8 | 0.99  |\n| SAEMark (N=10) | 99.2 | 96.8 | 98.0 | 0.99  |\n| SAEMark (N=20) | 98.7 | 98.7 | 98.7 | 0.99  |\n| SAEMark (N=50) | 99.7 | 99.8 | 99.7 | 1.0   |\n\nThis efficiency stems from typical use cases involving multiple textual units—we perform rejection sampling across sentences/paragraphs rather than individual tokens. \n\nAlthough our theoretical analysis are based on an i.i.d assumption of responses, the empirical results are done with top-k sampling from package text-generation-inference so the responses does not strictly i.i.d but still different. Our theoretical analysis is a worst-case scenario and the empirical results are actually significantly better.\nIn this case, we are technically not wasting the KV cache, and inference techniques like continous batching, concurrency are already supported.\n\n\n\n**W3: Novelty Presentation & Related Work**\n\nYou make an excellent point about clarity in presenting our contributions. You're absolutely right that we should acknowledge upfront that we apply existing Gemma Scope SAEs rather than proposing novel sparse autoencoder architectures. Our contribution lies in recognizing that **SAE features provide an ideal basis for post-hoc watermarking** and developing the complete framework around this insight.\n\nOur work represents **a fundamental paradigm shift**: we achieve training-free, logit-free, multi-bit watermarking that generalizes across natural languages and code and achieve SOTA performance compared with single-bit and multi-bit watermarks. While previous black-box methods like [1,2] you mentioned exist, they still require LLM paraphrasers or modify the sampling process. [2] is a similar approach to Waterfall which is in our experiments. Our approach uniquely operates on completely unmodified LLM outputs, requiring only rejection sampling based on deterministic features.\n\nWe'll add these citations and clarify that our innovation is the watermarking framework, not just the SAE architecture itself.\n\n**Q1: JumpReLU Parameters**\n\nJumpReLU in Gemma Scope uses pre-configured parameters (top-k sparsity) optimized during SAE training. We employ these pre-trained models without modification, as they already effectively capture semantic features across languages.\n\n**Q2: Key Terminology (k_priv vs k_pub)**\n\nYou've identified an important clarification need. Indeed, k_pub is used for both generation and detection—this is a symmetric scheme where the same public key enables both watermark embedding and detection. The \"priv/pub\" terminology reflects access patterns: k_priv remains secret with the user, while k_pub can be shared publicly for detection. We'll revise this notation to prevent confusion about asymmetric cryptography implications.\n\n**Q3: \"Backbone Model\" Clarification**\n\nApologies for the ambiguity. Qwen-2.5-7B-Instruct is the target LLM whose outputs we watermark (it generates the candidate texts). Gemma-2B serves as the compact Anchor LLM for feature extraction through its associated SAE. This separation enables watermarking any LLM's output while using consistent feature extraction. In our rebuttal to reviewer R51M we include new experiments on different target models and a new text-quality related dataset (BIGGen-Bench): \n\n| Model                 | Unwatermarked | SAEMark (Ours) | KGW  | Waterfall |\n| --------------------- | ------------- | -------------- | ---- | --------- |\n| `Qwen/Qwen2.5-7B-Instruct`   | **4.13**      | 4.05           | 3.97 | 4.02      |\n| `meta-llama/Llama-3.2-3B-Instruct` | 3.69          | **3.85**       | 3.56 | 3.62      |\n| `google/gemma-3-4b-it`             | **4.26**      | 4.23           | 3.98 | 4.19      |\n\n*(Scores on 5-point Likert scale, higher is better, Judged by the officially released model from BIGGen-Bench)*\n\nSAEMark consistently achieves the highest quality among watermarking methods, with minimal degradation from unwatermarked text, regardless of the target model.\n\n\n\n\n\n\n\nWhile rejection sampling introduces overhead, our results demonstrate that SAEMark achieves superior performance given same compute budget compared to existing methods. More fundamentally, we provide the best solution combining multiple critical capabilities: **training-free deployment, logit-free operation, multi-bit attribution, and cross-domain generalization**. The framework's modularity means it will automatically improve as better SAEs become available from the interpretability community.\n\nWould these clarifications and performance results under equal computational budgets address your concerns about practical viability? We believe the unique combination of capabilities, coupled with competitive real-world performance, represents a significant contribution despite the computational tradeoffs."}, {"review_id": "bfrLRe9dt0", "reviewer": "Reviewer_R51M", "summary": "This paper presents a new algorithm for text watermarking using sparse autoencoders. The key insight in the work is that different LLM samples to the same prompt exhibit vastly different features in a sparse autoencoder's space (SAE). Hence, via rejection sampling, an LLM sample can be chosen which closely matches the SAE statistic pre-chosen by a private key.\n\nThe authors perform several experiments on english/chinese/code output LLM detection and notice the proposed method outperforms several single and multi-bit watermarking baselines.", "questions": "Where exactly is the multilingual inductive bias in the method?\n\nWhat is the exact difference between single and multi-bit watermarks? Would be good to highlight this early in the paper, it wasn't super clear to me.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "*Strengths*\n\n- The paper studies a research topic of growing importance every since the introduction of ChatGPT --- automatic watermarking algorithms for detecting AI-generated text.\n\n- Unlike many previous watermarking methods, this method is post-hoc and does not modify the LLM generation process / logits. The rejection sampling ensures that all samples chosen were unperturbed LLM samples, ensuring high quality.\n\n- The paper is generally well presented and easy to read.\n\n- The proposed method outperforms several baseline watermarking algorithms in detection accuracy, while being better at preserving text quality (as judged by GPT-4o). The authors also include interesting experiments studying the robustness of their method to paraphrasing, ablations, and user-specific attribution.\n\n*Weaknesses*\n\n- Rejection sampling may not be suitable in all circumstances. For example, in lower entropy tasks (like factuality prompts), I would imagine the set of samples being produced across attempts are quite similar. How similar are the SAE / FCS features in this case? Does minor rewording the samples lead to significant changes in SAE space? Also, rejection sampling has its disadvantages. This will not work if the users set a low temperature in the models (which is common for some tasks), and it will kill any kind of LLM sample diversity for the user (which a user often desires).\n\n- A lot more work should be done on the text quality side of experiments. The paper currently only includes one experiment on text quality (PandaLM), which is not commonly used by modern LLMs in 2025 releases. Perhaps quality analysis could be done with/without watermarking on the BigGen Bench which has a wide variety of tasks (https://arxiv.org/abs/2406.05761)? Even performance differences on reasoning benchmarks like MATH-500 or MMLU will be very helpful to see before/after watermarking.\n\n- Minor, but the proposed method is quite complex. Is there no simpler method to derive discriminative features from SAE vectors, rather than deriving feature concentration scores? FCS scores do seem a critical part of the paper and should be properly described in the main body in my opinion. I'm especially confused about how exactly the target FCS determined from a random seed public key, and how do you ensure it's within the range of FCS across LLM samples?\n\nMinor, but consider comparing against another post-hoc watermarking method introduced last year: https://arxiv.org/abs/2406.14517", "was_revised": true, "final_justification": "Thanks to the authors for the detailed rebuttal and extra experiments. I will continue to support acceptance for the paper, but will not be able to raise my score further since my concerns about rejection sampling  (also voiced by other reviewers) + complexity remain.\n\nAlso I appreciate the BigGen experiments and they should be added to the paper. Evaluation on MATH/coding benchmarks will be very interesting -- these are low entropy tasks where there could be clear losses with this kind of repeated sampling.", "rebuttal": "Thank you for your thoughtful and constructive review! We greatly appreciate your recognition of our work's originality and significance. Your positive assessment of our post-hoc approach and detailed feedback helps us strengthen the paper significantly.\n\n\n\n**Q1: Where exactly is the multilingual inductive bias in the method?**\n\nThe multilingual capability in our method comes from the fact that we don't have any language-specific components or inductive biases—it's actually the *absence* of language-specific design that enables universal operation. Unlike previous methods that require language-specific components (KGW needs language-specific tokenizers, SWEET uses English syntax patterns, semantic watermarks need language-specific paraphrasers), **we only define unit boundaries (sentences for text, functions for code)**. **Everything else—feature extraction, FCS calculation, rejection sampling—operates identically across languages**. This domain-agnostic design is why we achieve 99.7% accuracy on English, 99.2% on Chinese, and 74.5% on code without any language-specific tuning.\n\n\n\n**Q2: What is the exact difference between single and multi-bit watermarks?**\n\nThank you for highlighting this—we'll clarify this distinction upfront. Single-bit watermarking embeds one bit of information, answering only \"is this AI-generated?\" Multi-bit watermarking embeds multiple bits to encode additional information—in our case, which specific user among thousands generated the text. This is crucial for accountability: single-bit methods detect but cannot attribute. *Multi-bit watermarking is significantly harder as we must preserve multiple bits through generation while maintaining quality*. Our reliable multi-bit watermarking represents a major advance in making LLM outputs truly accountable. \n\nAlthough injecting 10 bits of information reliably seems modest, it is already state-of-the-art performance and this is indeed a challenging field.\n\n\n**W1: Rejection sampling limitations in low-entropy tasks**\n\nYou raise an excellent point. For low-entropy tasks, we acknowledge the challenges. We now include additional analysis on TruthfulQA, where we collect responses and analyze the FCS score distribution, results show that factual QA responses exhibit lower FCS variance (0.017 vs 0.029) but still following a normal distribution, making watermark embedding harder but our theoretical framework still applies. This explains the performance degradation on constrained tasks like code generation (66.3% on MBPP).\n\nFor deterministic settings (temperature=0), we can control the random seed based on the watermark key, making different keys produce different deterministic outputs. This maintains both reproducibility and watermark capability without requiring temperature sampling. We'll clarify these limitations and solutions in the revision.\n\n\n\n**W2: More comprehensive text quality evaluation**\n\nWe completely agree and thank you for the BigGen-Bench suggestion! We conducted extensive new experiments across three models to comprehensively evaluate quality preservation:\n\n| Model                 | Unwatermarked | SAEMark (Ours) | KGW  | Waterfall |\n| --------------------- | ------------- | -------------- | ---- | --------- |\n| `Qwen/Qwen2.5-7B-Instruct`   | **4.13**      | 4.05           | 3.97 | 4.02      |\n| `meta-llama/Llama-3.2-3B-Instruct` | 3.69          | **3.85**       | 3.56 | 3.62      |\n| `google/gemma-3-4b-it`             | **4.26**      | 4.23           | 3.98 | 4.19      |\n\n*(Scores on 5-point Likert scale, higher is better, Judged by the officially released model from BIGGen-Bench)*\n\nSAEMark consistently achieves the highest quality among watermarking methods, with minimal degradation from unwatermarked text. Additionally, when controlling for equal token generation budgets, SAEMark achieves the highest detection accuracy, demonstrating superior performance-per-compute.\n\n\n\n**W3: FCS complexity and method description**\n\nYou're absolutely right about needing clearer explanation. The target FCS for each segment is deterministically derived using the public key as a seed to sample from the observed distribution N(0.142, 0.029²), constrained to [μ-2σ, μ+2σ] for feasibility.\n\nMore importantly, as you intuited, FCS is just one instantiation of our general framework. Any deterministic feature with natural variation could work. The core innovation is the rejection sampling framework for post-hoc watermarking, not the specific feature choice. We'll move FCS details to the main text and emphasize this generality.\n\n**Additional comparison**\n\nThank you for suggesting the 2024 post-hoc work. This work is actually a simpler variant of Waterfall as both require a LLM paraphraser to inject watermark information by paraphrasing. We'll include a detailed comparison, noting that while it requires modifying the sampling process, our approach operates on completely unmodified LLM outputs.\n\n\nWe're grateful for your constructive feedback and recognition of our originality. The BigGen-Bench results confirm that our method achieves superior performance while genuinely preserving output qualities. Combined with our multi-bit attribution capability and domain-agnostic design, we believe these improvements address your concerns comprehensively.\n\nWould these additions—especially the extensive quality evaluation and clearer explanations—be sufficient to demonstrate that our novel approach deserves strong support? We're committed to making this contribution as clear and well-evaluated as possible."}, {"review_id": "zorUo9vGYy", "reviewer": "Reviewer_SdYH", "summary": "This paper introduces SAEMARK, a watermarking method for LLMs that embeds personalized watermarks without requiring access to model logits or parameters. The key innovation is using Sparse Autoencoders to extract semantic features from generated text and then applying rejection sampling to select outputs whose feature concentrations match watermark-specific target distributions derived from user keys.", "questions": "- Could you clarify the distinction between your claimed ``multi-bit watermarking'' and what appears to be single-bit detection followed by user identification? How do you justify calling this multi-bit when no actual multi-bit information is extracted from individual watermarked texts?\n\n- Given that your method supports only 1,200+ users, how do you envision scaling this approach to real-world deployment scenarios requiring millions of users? What are the theoretical and practical limits of your user identification approach?\n\n- Your code generation results show only 66.3% F1 on MBPP, which is marginally better than traditional methods. Why do you claim effectiveness on code generation when the performance appears significantly inferior to your natural language results?\n\n- Given that SAE features are likely to change substantially under semantic-preserving paraphrasing, how confident are you that your approach maintains robustness against such attacks? Can you provide theoretical or empirical justification for this robustness?", "limitations": "Please refer to the weaknesses.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strength:\nEnabling watermark injection without requiring access to model logits, which is a long-discussed issue.\n\nWeaknesses:\n1. The paper claims to be a ''multi-bit watermarking'' method, but this is fundamentally misleading. The approach only supports 1,200+ users, which is not a particularly large scale for real-world deployment. More importantly, the method does not actually extract multi-bit information from the watermarked text. Instead, it performs user matching by comparing observed feature patterns against different user-specific targets. This is essentially a single-bit detection problem (watermarked vs. non-watermarked) followed by user identification, rather than true multi-bit information extraction. The authors conflate user attribution with multi-bit watermarking, which are conceptually different tasks.\n\n2. While the authors emphasize multilingual capabilities, many existing baselines (e.g., KGW) already support multiple languages effectively. The paper fails to clearly articulate what specific limitations existing methods have in multilingual settings that SAEMARK addresses. Furthermore, although the authors claim effectiveness on code generation, the experimental results show very limited performance on programming languages (66.3% F1 on MBPP), which is barely better than some traditional methods and significantly worse than their performance on natural language tasks.\n\n3. The robustness experiments are concerningly limited, testing only up to 15% word deletion and synonym substitution attacks. Critically, the paper completely omits evaluation against paraphrasing and rewriting attacks, which are among the most practical and effective ways to remove watermarks. Many existing watermarking methods have specifically designed defenses against such attacks. The absence of these evaluations raises suspicions about whether the SAE-based approach is fundamentally vulnerable to content rewriting, since SAE features are likely to change significantly when sentences are paraphrased or rewritten, even if semantic meaning is preserved.\n\n4. The core contribution appears to be simply applying existing SAE technology to watermarking, which represents limited technical innovation. The method seems relatively crude - essentially computing feature concentration scores and using rejection sampling to match target distributions. The approach lacks theoretical depth and appears more like an engineering application of existing interpretability tools rather than a fundamental advancement in watermarking techniques. The reliance on external SAE models also introduces additional dependencies and potential points of failure that are not thoroughly analyzed.\n\n5. The experimental evaluation lacks comprehensive comparison with state-of-the-art robust watermarking methods, particularly those designed to withstand semantic-preserving attacks. The paper's focus on detection accuracy metrics while avoiding the most challenging attack scenarios suggests the evaluation may be cherry-picked to favor the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. While we see you have several significant concerns, we believe they may stem from a few *key misunderstandings and factual errors* regarding our method's contributions, novelty, and the experiments. We appreciate this opportunity to clarify these points and present new experimental results that we are confident will strengthen your evaluation of our work.\n\n**W1, Q1, Q2: Misunderstanding on Multi-bit Watermarking and Scalability**\n\nYou pointed out that our method is \"fundamentally misleading\" by conflating user attribution with multi-bit watermarking and that supporting 1,200+ users is insufficient. **This reflects a core misunderstanding: Embedding a unique identifier for one of N users into watermarked text is, by definition, equivalent to embedding $log₂(N)$ bits of information into the text. This is the central challenge of multi-bit watermarking.**\n\nOur experiments, conducted under a **controlled text length**, show that SAEMark reliably injects **10+ bits** of information (corresponding to 1,024+ users with >90% accuracy ). **This is already state-of-the-art performance for a training-free multi-bit watermark**, significantly outperforming existing methods like Waterfall (which is a recent paraphrase-based multi-bit SoTA watermark), as shown in Figure 4 in our paper. The number of users we support scales directly with the length of the generated text, a universal property of all watermarking schemes. Clearly it is not possible to inject infinite extra information into texts of limited length.\n\n\n\nWe add new experiments showing that when controlling for the **same total token generation budget**, which is the actual performance bottleneck, SAEMark is not only accurate (>99% F1) but also significantly faster than leading methods.\n\n| **C4**        | Acc.     | Rec.     | F1       | Latency |\n| ------------- | -------- | -------- | -------- | ------- |\n| KGW           | 98.8     | 99.2     | 98.8     | 174s    |\n| UPV           | 86.1     | 74.2     | 84.2     | 94s     |\n| DIP           | 99.3     | **99.8** | 99.3     | 85s     |\n| Waterfall     | 98.7     | 97.0     | 97.9     | 52s     |\n| SAEMark(N=50) | **99.7** | **99.8** | **99.7** | **40s** |\n\n\n\n\n| **LCSTS**     | Acc      | Rec      | F1       | Latency |\n| ------------- | -------- | -------- | -------- | ------- |\n| KGW           | 99.2     | 99.8     | 99.0     | 85s     |\n| UPV           | 94.5     | 98.4     | 94.7     | 94s     |\n| DIP           | **99.6** | **99.6** | **99.6** | 178s    |\n| Waterfall     | 98.8     | 97.6     | 98.2     | **33s** |\n| SAEMark(N=50) | 99.2     | **99.6** | 99.2     | 40s     |\n\n\n\nAdditionally, although in our theoretical analysis we could achieve 99%+ accuracy with N=50 per unit, our empirical results yield strong performance at N=10 already. **Using only 20% of compute overhead is sufficient to produce empirically strong performance**, this is because typical scenarios include multiple units instead of only one (typical LLM calls would generate multiple sentences). We now report single-bit watermarking performance with SAEMark given different sets of N (which is the most significant factor for compute cost).\n\n| C4             | Acc  | Rec  | F1   | AUROC |\n| -------------- | ---- | ---- | ---- | ----- |\n| SAEMark (N=5)  | 98.7 | 77.4 | 86.8 | 0.99  |\n| SAEMark (N=10) | 99.2 | 96.8 | 98.0 | 0.99  |\n| SAEMark (N=20) | 98.7 | 98.7 | 98.7 | 0.99  |\n| SAEMark (N=50) | 99.7 | 99.8 | 99.7 | 1.0   |\n\n| LCSTS          | Acc  | Rec  | F1   | AUROC |\n| -------------- | ---- | ---- | ---- | ----- |\n| SAEMark (N=5)  | 98.6 | 72.6 | 83.6 | 0.98  |\n| SAEMark (N=10) | 99.0 | 96.0 | 97.5 | 0.99  |\n| SAEMark (N=20) | 98.6 | 98.0 | 98.6 | 0.99  |\n| SAEMark (N=50) | 99.2 | 99.6 | 99.2 | 1.0   |\n\n**W2, Q3: Multilingual and Code Generation Effectiveness**\n\nYou claim our multilingual capabilities are not novel and that our code generation performance is \"very limited.\" This is incorrect. Our method’s multilingual strength lies not in a specific component but in its complete **absence of language-specific design**. Unlike methods like KGW (which requires language-specific selection of red-list or green-list tokens) or paraphrase-based watermarks that require language-specific paraphraser or parsing modules, our approach is truly domain-agnostic. \n\nTake KGW as an example: KGW randomly selects green-list tokens based on a seed, however since modern LLMs are multilingual, if the user wants to inject watermarks into English texts but without any constraints, it would be likely some tokens from another language are chosen and would impact quality. In contrast we do not need such manual intervention, and this is why we achieve SOTA performance on both English and Chinese without any modification. \n\nRegarding code generation, it is a known challenging, low-entropy domain where all watermarking methods struggle. Acknowledging this difficulty is key. Our 74.5% accuracy (66.3% F1) on MBPP is not \"barely better\"; it is **state-of-the-art performance** that significantly outperforms all baselines, including specialized code watermarkers like SWEET (`Who Wrote this Code? Watermarking for Code Generation`, 72.4% accuracy, 45.9% F1).\n\nTo further support our quality claims, we followed the suggestion of Reviewer R51M and ran new tests on the BIGGen-Bench, a multilingual text quality benchmark, which confirm SAEMark preserves quality best.\n\n| Model    | Unwatermarked | SAEMark (Ours) | KGW  | Waterfall |\n| -- | ------ | ---- | ---- | --------- |\n| `Qwen/Qwen2.5-7B-Instruct`   | **4.13**      | 4.05           | 3.97 | 4.02      |\n| `meta-llama/Llama-3.2-3B-Instruct` | 3.69          | **3.85**       | 3.56 | 3.62      |\n| `google/gemma-3-4b-it`             | **4.26**      | 4.23           | 3.98 | 4.19      |\n\n*(Scores on 5-point Likert scale, higher is better, Judged by the officially released model from BIGGen-Bench)*\n\nSAEMark consistently achieves the highest quality among watermarking methods, with minimal degradation from unwatermarked text. Additionally, when controlling for equal token generation budgets, SAEMark achieves the highest detection accuracy, demonstrating superior performance-per-compute.\n\n\n\n**W3, Q4: Robustness Against Paraphrasing and Rewriting Attacks**\n\nThank you for emphasizing the critical importance of robustness, however there's an important misunderstanding: To improve readability, we report attack ratios ranging from 5% to 15% of 3 different attacks in Figure 5 of our paper and it already include 10 distinct ROC curves. **We actually have a dedicated section Appendix E, regarding adversarial robustness evaluation and Figure 9 to 11 shows attack ratios from 10% to 50% and our method still exhibit strong performance**, even in extreme scenarios like CTXSubstitution with $\\epsilon=0.5$.\n\nThe original paper already demonstrated strong resilience. We now add an extensive new experiments against sentence-level paraphrasing and extended our context-aware substitution attacks. The results demonstrate strong robustness. Similar to current paraphrasing watermark settings, we use a LLM paraphraser to randomly paraphrase parts of the text.\n\n| Type                | 10%  | 20%  | 30%  | 40%  | 50%  |\n| ------------------- | ---- | ---- | ---- | ---- | ---- |\n| **CTXSubstitution** | 0.98 | 0.95 | 0.89 | 0.85 | 0.82 |\n| **Sent Paraphrase** | 0.99 | 0.96 | 0.92 | 0.86 | 0.80 |\n\nEven when **50% of the sentences are paraphrased**, our method maintains a high AUC of 0.80. At a 50% context-aware substitution rate, the AUC is still 0.82. This directly proves that our method, which relies on deeper semantic features captured by SAEs, is fundamentally resilient to strong attacks.\n\n\n\n**W4: Novelty and Technical Contribution**\n\nYou characterized our contribution as \"simply applying existing SAE technology.\" We see how our paper might have given that impression, and we were perhaps too modest in presentation.  Our core innovation is a **fundamental paradigm shift in watermarking**: we propose **a general, training-free, logit-free framework based on rejection sampling**. Our theoretical analysis in Section 3.2 is novel, providing a clear and generally applicable foundation. **SAEs are merely one powerful instantiation of this framework. You could replace the SAE with *any* deterministic feature extractor whose features vary during generation and follow a normal-like distribution, and our theoretical guarantees would still hold.** This establishes a new, flexible, and powerful direction for post-hoc watermarking.\n\n\n\n**W5: Selection of baselines**\n\nWe take your concern about \"cherry-picking\" baselines very seriously and want to clarify our evaluation process. We have a dedicated section on our selection of baseline methods in **Appendix C (from page 15 to 18), including introduction to every single baseline**. \n\nTo ensure fairness and completeness, we used the standard **MarkLLM toolkit** , which implements numerous SOTA watermarking methods, and we reported results against all of them in Table 2. For our multi-bit experiments, we selected Waterfall as the primary baseline for clear and principled reasons: it is **(1) the current state-of-the-art multi-bit watermark , (2) open-source , and (3) training-free**, making it the most direct and fair competitor to our approach. **There was no cherry-picking.** **If there's any specific open-source watermarking method that you believe we have missed, we are happy to add new experiments during discussion period.**\n\nWe trust these clarifications and the substantial new experimental evidence have thoroughly addressed your concerns. **SAEMark achieves what no previous method has: a training-free, logit-free, multi-bit, and robustly generalizable watermarking solution.**\n\n**We believe this is a significant and practical contribution to the field and respectfully ask you to reconsider your score in light of this new information.**"}]

Author rebuttal:

Meta-review: This paper proposes SAEMark, a novel post-hoc watermarking framework for LLMs that embeds personalized multi-bit watermarks using sparse autoencoders and rejection sampling, without requiring access to model logits or weights. The reviewers generally agreed that the problem is timely and important, and that the method is technically sound, compatible with API-based models, and evaluated across multiple languages and domains including code. Two reviewers (wvir, R51M) were convinced by the rebuttal and discussions, highlighting the method’s originality, robustness to paraphrasing, and strong performance-per-compute, and improved their recommendations to accept/weak accept. Reviewer JSXp maintained a borderline accept after acknowledging clarifications, while Reviewer SdYH remained strongly negative, questioning the conceptual framing of “multi-bit watermarking,” the robustness evaluation, and the degree of novelty. The authors provided extensive new experiments (on BigGen-Bench, paraphrasing robustness, efficiency analysis) and clarified methodological details, which largely satisfied the engaged reviewers but did not change SdYH’s position. On balance, I believe the paper is ok to be published, as long as the authors incorporate all changes (including the promised ones) in their final version.
Decision: Accept (poster)