Robust Distortion-Free Watermark for Autoregressive Audio Generation Models
Authors: Yihan Wu
Georgios Milis
Ruibo Chen
Heng Huang
Venue: NeurIPS 2025 poster ()
Keywords: Audio watermarking
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: The rapid advancement of next-token-prediction models has led to widespread adoption across modalities, enabling the creation of realistic synthetic media. In the audio domain, while autoregressive speech models have propelled conversational interactions forward, the potential for misuse, such as impersonation in phishing schemes or crafting misleading speech recordings, has also increased. Security measures such as watermarking have thus become essential to ensuring the authenticity of digital media. Traditional statistical watermarking methods used for autoregressive language models face challenges when applied to autoregressive audio models, due to the inevitable ``retokenization mismatch'' - the discrepancy between original and retokenized discrete audio token sequences. To address this, we introduce Aligned-IS, a novel, distortion-free watermark, specifically crafted for audio generation models. This technique utilizes a clustering approach that treats tokens within the same cluster equivalently, effectively countering the retokenization mismatch issue. Our comprehensive testing on prevalent audio generation platforms demonstrates that Aligned-IS not only preserves the quality of generated audio but also significantly improves the watermark detectability compared to the state-of-the-art distortion-free watermarking adaptations, establishing a new benchmark in secure audio technology applications.

Robust Distortion-Free Watermark for Autoregressive Audio Generation Models
Yihan Wu; Georgios Milis", Ruibo Chen', Heng Huang Department of Computer Science University of Maryland, College Park fywu42, milis, rbchen, heng) @umd edu
Abstract
The rapid advancement of next-token prediction models has led to widespread adoption across modalities, enabling the creation of realistic synthetic media. In the audio domain, while autoregressive speech models have propelled conversational interactions forward, the potential for misuse, such impersonation phishing schemes crafting misleading speech recordings, has also increased. Security measures such as watermarking have thus become essential ensuring the au- thenticity of digital media. Traditional statistical watermarking methods used for autoregressive language models face challenges when applied to autoregressive audio models, due to the inevitable "retokenization mismatch". the discrepancy between original and retokenized discrete audio token sequences. To address this, introduce ALIGNED- -S, novel, distortion-free watermark, specifically crafted for audio generation models. This technique utilizes a clustering approach that treats tokens within the same cluster equivalently, effectively countering the retokenization mismatch issuc. Our comprchensive testing on prevalent audio generation platforms demonstrates that ALIGNED-IS not only preserves the quality of generated audio but also significantly improves the watermark detectability com- pared to the state-of-the-art distortion-fiet watermarking adaptations, establishing a new benchmark in secure audio technology applications. We release the code in cwm.com////////. g- mimiila lis/ AlignedIS
rext-token-prediction
1 Introduction
Autoregressive audio generation models (Lakhotia et al. 202 Rubenstein al. 2023 Borsos et al.
2022 Zhang et al. 2023 Nguyen et al. 2025 Zhan et al. 2024 Ge et al.] 2023 Lu et al. 2024), such
as those enabling sophisticated voice synthesis, have significantly advanced in mimicking human-like speech. As these models become integral to various applications, from virtual assistants to real-tine translation, their potential misuse also escalates. Malicious uses include impersonating individuals in phishing attacks, fabricating audio for misinformation, and automating scam calls with natural- sounding voices. Additionally, the spread of synthetic audio can undermine the authenticity of digital communication and pose challenges in legal contexts where recording verification is crucial. To address these concerns, implementing robust and detectable watermarks in synthetic audio becomes essential, ensuring traceability and accountability in the use of generative models while safeguarding against their unauthorized exploitation.
Statistical watermarking techniques promising method to identify machine-generater content
from autorcgressive language models (Kirchenbauer et al. 2023 Liu & Bu 2024 Chen et al. 2025a).
"qqual contribution
This work was partially supported by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS. 2347617, RISE 2536663.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, due to retokenization mismatch (Wu et al. 2025). directly applying them to audio generation models leads to poor detectability. Unlike autoregressive language models, where text tokenization is deterministic and reversible, autoregressive audio models incorporate an additional encoder and decoder that map from audio to discrete tokens and back. In the process of audio generation, an audio prompt is encoded into a sequence of tokens. Subsequently, this sequence undergoes next-token prediction to generate an output sequence, which is then decoded back into audio form. Watermark detection requires re-encoding the generated audio into its tokenized form. However, this retokenized token sequence does not exactly match the original in-generation sequence, a phenomenon we refer to as retokenization mismatch.
To tackle this challenge, we introduce ALIGNED-IS, a novel, robust, and distortion-fre watermark specifically designed for audio generation models. We observe that mismatch arises from different discretization of similar continuous features, which should be close in the encoder and decoder's s feature spaces. Leveraging this insight, we developed clustering- based watermarking framework that considers tokens within the same cluster as equivalent. We summarize our contributions as follows:
We develop ALIGNED-IS, the first distortion-free watermark for autoregressive audio generation models. We identify the retokenization mismatch phenomenon and we propose a novel clustering based distortion- watermarking algorithms address this challenge.
Through comprehensive experiments, we validate the iistortion-freenees detectability, and robustness of ALIGNED-IS on popular open-source audio generation models. Our results show significantly improvement in detectability compared to directly applying existing distortion-free watermarks to audio generation models.
2
Related Work
Audio generation models. With the success of large language models, researchers have devel- oped multimodal foundation models that extend transformers to handle continuous signals through
modality-speciiff encoders and decoders (Vaswani al. 2017). Lakhotia
al. (202T) first re-
framed pure audio generation as a language-modeling task (Lakhotia et al.) 2021). which gave rise to AudioLM (Borsos al., 2022) and later Audi integration of text generation capabilities (Rubenstein et al. (023). Building on this, SpeechGPT (Zhang et al. 2023) and SpiritLM (Nguyen) et 2025) discretize HuBERT features (Hsu al.] (021) into semantic units that fine-tuned LLaMA (Touvron et al.) 2023) consumes as an expanded vocabulary of text and audio tokens. The same discretization strategy supports fine-grained tasks such as voice cloning from text and discretized voice prompt (Chen et al., 2024b). Other models-including SEED-LLaMA (Ge et al. 2023). Unified-IO (Lu et al. a 2024), and AnyGPT (Zhan al. repeeste audio and images with discrete token sequences alongside text. In contrast, CoDi-2 (Tang et al. (002)), VITA (Fu al. 2024), and NEXT-GPT (Wu et al.. (024) employ a transformer decoder that directly processes continuous feature vectors together with text token embeddings, treating multimodal outputs as regression while still using next-token prediction for text.
Post hoc audio watermarking, Embedding watermarks into host audio dates back decades (Li &
Chang 2006), leveraging human insensitivity to mid-frequency bands. More recently, learnng methods use autoencoder architectures to invisibly encode payloads into a frequency transform of the signal (Liu et al.) 2024a] Chen et al.) (003).. The work Sf Roman al.) (2024) further introduces temporally localized watermarking with time-dependent detection, and error- codes have been employed to boost robustness (Wu et al., (203a).. However, these approaches degrade output quality and offer no formal statistical guarantee for reliable watermark detection. Furthermore, they are fragile to waveform perturbations, and require an additional processing step to embed them.
Distortion-free watermark. Aaronson 2022) introduced a pioneering distortion free watermarking approach that utilizes Gumbel-Softmax to alter token distributions. [Christ et al. (2033) and Kuditipudi et al. (023) applied inverse-sampling and Gumbel-Softmax, respectively, to modify the token distributions watermarked content, employing watermark keys based on either token positioning or predetermined key lists. However, the technique from Christ et al. (002)) exhibits limited resilience when altered and lacks empirical evidence of its detectability. In contrast, Kuditipudi et al.) (2023)'s 's method demands extensive resampling from the secret key distribution for detection, proving

---

Watemarking
1.0
Row sequeric Ralaxenized
Prompt
Tokens
Detector
Text
Result
Audio Wattermrking
Decoder
Audio
0.4
Generator
Prompt
Tokens
wavform
Detector
Audio
Encoder
Takens
Result
wavform
E
KOW/
y reweight
Figure 1: Left: Illustration of the retokenization mismatch problem during audio watermark detection. Right: Performance comparison of audio watermark detection on raw versus retokenized audio sequences. We evaluate KGW watermark (Kirchenbauer al., 2023) DiPmark (Wu et al., 2023b), and 7-reweight (Hu et al. 2023). reporting the true positive rate at a fixed 1% false positive rate.
inefficient for extensive texts. Hu et al. (023) proposed inverse inverse-sampling and reweight strategics for watermarking, though their detection method is not model-agnostic and requires access to the language model API and specific prompts. Wu et al. (2023b) refined the reweight technique and introduced model-agnostic detection mechanism. Dathathri al. (2024) proposed SynthID, which enables distortion-frrennes of LM watermarking with multiple generations.
3 Preliminary
The vocabulary (or token) is denoted by V and its cardinality by N == V We define the set V, which includes all possible token sequences including those of zero length and the set A, which includes all possible audios. Within autoregressive audio generation model, a token sequence is generated based on a specific prompt. At any given step, the probability of producing the next token In+1 V, given the preceding sequence Ti...., En is denoted by PM (In+1 æ1; ... J.)) For simplicity and clarity, we adopt a more important to note that the prompt is intentionally excluded from this notation. In audio generation models, we denote the audio-token encoder by E(-) A + V and the ooken-audio decoder, or vocoder, by D(-): - A.
Notations. We follow the notations used in (Hu al. 2023
concise notation: PMr 2I x1:n), where #n-ln+m
(In+1.... Zn+m).
3.1 Statistical Watermarks
In watermarking applications, the service provider employs set of i.i.d. watermark codes {0, O, e N). defined over the code space θ. Each code 0, is typically derived from a secret key key K. and the n-gram preceding context, denoted 2t-n:t-1.
In the watermark generator, reweight strategy is used embed statistical signal into the generated content. Let P denote the set of all probability distributions over the token set V. The reweight strategy is function Pw x - P. For the token distribution the (n + 1)-th generation step, ( II..) E P. the watermarked distribution is defined by Pw Pw(Pas(zu (Prr( [I..), 0,). For brevity, this represented as (w 2i:, 0.). A distortion-free watermark ensures that the averaged distribution Pw (#n+1 Z1:m 02) with respect to 0, is equal to the original distribution PM (1n+11 I.m))
Definition 3.1 (Distortion-fret watermark). Given the watermark code set θ, a distribution Pe on O, original LM distribution Par and the watermarked distribution Pw (-10 +) A distortion-free watermark should satisfy Vr E V,
Eaw (P Fimm 0)] = (M #1m.).
Current popular distortion strategies include Gumbel-softma (Aaronson 2022 inverse- sampling (Christetal, 2023
2023] Huu et [2023] Kuditipudi et al. (023) and reweight-based strategy (wul
al. 2023b Dathathri et al. 2024 Feng al. 2025 Chen et al. 2025a).

---

During watermark detection, the user only has access to the watermark key, the reweight strategy, and the generated audio. The detector employs a hypothesis testing approach to ascertain the presence of the watermark signal. The null hypothesis Ho is defined as "The content is generated without the presence of watermarks". The detector adopts score function based on the watermark key and the reweight strategy, which exhibits statistical bias between the watermarked and unwatermarked token sequences.
3.2 Autoregressive audio generation models
Autoregressive audio models typically begin by encoding raw waveforms into self-supervised feature representations, such as those produced by wav2vec (Baevski et al. (020) or HuBERT (Hsu et al.) 2021). These continuous features are then quantized into discrete semantic units. In contrast to text- where tokens generally correspond reversible character or subword units (Sennrich al.) - audio waveform is segmented into overlapping frames and processed by a feature extractor. To discretize the feature space, a clustering algorithm (c.g., k-means) partitions it into N,, clusters, with each frame assigned to the nearest cluster centroid. The resulting sequence of centroid indices serves as "audio tokens" for the language model. During inference, these discrete tokens are transformed back into a waveform using a neural vocoder (Kong et al. (2020). In multimodal text text-audio LLMs. the overall vocabulary size is N No. + Ne. where N, denotes the number of text tokens (Zhang et al. 2023 Nguyen al. 2025
Retokenization mismatch. Let denote the token sequence output by the audio generation model, where the final audio is obtained by decoding through the token-audio decoder D(x),, referred to as vocoder. For watermark detection, the generated audio is passed through the audio-token encoder E(D(a)), and a hypothesis test is conducted on ((D(x)) to identify the presence of a watermark signal. Nonetheless, (()():) often differs from the original x, which weakens the statistical evidence captured by the detection scores. This discrepancy can be viewed as an unavoidable token- level perturbation occurring during the detection process. As shown in Figure [ there a large gap between the detectability of the raw token sequence æ and the retokenized sequence (()(x)).
Methodology
To address the retokenization mismatch issue autoregressive audio generation models, we developed ALIGNED-IS, IS, clustering-based watermarking framework. In ALIGNED-IS, audio tokens are initially segmented into clusters. Subsequently, we introduce a novel, distortion free reweight strategy, aligned inverse sampling, tailored specifically for these clustered tokens. During watermark generation, this reweight strategy alters the output distribution according to the clusters. For watermark detection, the detector verifies whether the current token corresponds to the expected cluster, as determined by the reweight strategy and the embedded watermark code. Since mismatched tokens are likely within the same cluster, the detector can still accurately capture the correct statistical signal, even in the presence of retokenization mismatches.
In this section, we first present the clustering method for tackling retokenization mismatch and distortion-free reweighting strategy, aligned inverse sampling, which specifically devised for the clustered tokens. Then, we introduce our general watermark algorithm and detection statistic.
4.1 Audio token clustering
During clustering, the objective is to split all audio tokens into distinct clusters based on their similarity. It is crucial to note that the clustering algorithm is executed only once for each audio generation model, its results are stored, and can be accessed during watermark generation and detection. Consequently, this approach does not increase the computational cost during sampling. To achieve this, we collect the audio token embeddings (e1, (.., Cm } based on the token-audin encoder E and use k- k-maans algorithm to generate the corresponding clusters {ci...., }. We employ k-means since the discretization process uses the Euclidean norm of audio feature vectors to discretize them, by selecting the nearest token embedding vector. While acessing the embedding vectors in open-source models trivial, closed source model provider can still implement our watermarking method and expose its detection API.

---

Table 1: The effect of clustering on retokenization mismatch across datasets for SpiritLM.
Dataset
Mismatch Rate Before
Mismatch Rate After
Reduction (%)
MMW Book Report
0.3749
0.2117
43,55%
MMW Story
0,3652
0,2174
40.47%
MMW Fake News
0,4295
0,2300
46.44%
Dolly CW
0,3634
0,2134
41.30%
Longform QA
0.3757
0,2109
43.85%
Finance QA
0.3587
0.2133
40.54%
With our segmentation strategy, mismatched tokens are more likely to be in the same cluster because they are supposed to share similar embeddings in the encoder E. The watermark generator and detec- tor can utilize the cluster information to avoid the detectability reduction caused by the retokenization mismatch. The inverse sampling watermark distortion-free method that can be applied directly to clustering scenarios.
To quantify the effectiveness of our clustering method in mitigating retokenization mismatch, report the token mismatch rates before and after applying clustering on SpiritI.M across multiple datasets in Table [ The mismatch rate is computed by comparing the original response tokens with those obtained after a decode-then-enevee clustering method significantly reduces the mismatch rate, demonstrating its effectiveness in aligning tokens during watermarking generation and detection.
retokenization process. As shown in the table, our
We have also evaluated other popular clustering methods, e.g., (Gaussian mixture models, spectral clustering) and found that they produce similar results. We did not explore methods beyond clustering, as it is the most natural and direct approach to address the retokenization mismatch. Empirically, clustering has proven effective in significantly reducing such mismatches.
4.2 Aligned inverse sampling
Inverse Sampling. Let Cl....,C,, represent the identified clusters, and let Pr(ci)..... Pr(cx)
denote the sum of the token probabilities within them, such that Pr(ct) Ezec, (x). It is natural to map these probabilities onto the interval [0,1] and employ inverse sampling to pseudo-randomly select number (0) [0, 1] seeded by the watermark code 8. The subsequent token is then sampled from cluster c; if falls within the interval EE= Pr(c,). Ei Pr(c,)], with Pr(co) == O. During detection, cluster information and the pseudo-random number r(0) can be reconstructed using the watermark code 0, allowing to compute statistical score by comparing the cluster against r()).
However, since token probabilities are unknown during watermark detection, we cannot ascer- tain the cluster probabilities Pr(ci)...., Pr(cs). This discrepancy leads to an alignment issue between r0) and the identified cluster c, it becomes uncertain whether r(0) is within the interval DE Pr(c;), Ejoo Pr(c)], To address this, Christ et al. y023); Kuditipudi et al. (2023) proposed a position-based statistical score for watermark detection. The underlying principle is that if r()) is close to then the index of the selected cluster during watermark generation is likely close to h (the end of the clusters). However, the detection methods proposed in Kuditipudi et al.) (2023) cannot provide theoretical guarantee of the false positive rate. Besides, this position-based score has shown low detectability, as evidenced in (uditiipudi et al. (2023) and our experimental findings. To address the flaws of inverse sampling, we introduce aligned inverse sampling, which substantially improves detection efficiency while providing provable guarantee on the false false-positive rate.
Aligned Inverse Sampling. Consider the scenario where Pr(c;) ... Pr(c,)= J. In this case,
the pseudo-random number and the identified cluster are aligned. Specifically, r()) E i-1 1 . the detector can confidently assert that 6; was generated through inverse sampling with r(0). A natural enhancement to inverse sampling involves rearranging the cluster probabilities within the interval [0, to emulate this aligned scenario.
Details on this method are in Algorithm 3
Theorem 4.1. Aligned inverse sampling is distortion-free watermark.
5

---

Algorithm ALIGNED-IS generator.
1: Input: secret key key, prompt 2 m-m generate length E N, token- decoder D.
2: Initialize watermark code history hist.
3: for i l..... do
4: Calculate the token distribution for generating the i-th token Par(. a x--mxi1)..
5: Generate a watermark code 0; (key, fi-nnii1).
6: if 0 E hist then
7:
Sample the next token z; using original distribution PM (M(-1 (-(--mmmm- else
8:
9:
Generate the pseudo-random number r(:)).
10:
Calculate watermarked distribution Pw Pw(-/a -1---mmm))))-- via aligned inverse sampling. Sample the next token H; using distribution Pw (mx-)-x)))..
11:
return audio waveform (xx1:)
Algorithm ALIGNED-IS detector.
1: Input: audio waveform a, audio-token encoder E, secret key key, score function 8, threshold z.
2: calculate token sequence V1:t ((a)
3: Initialize the score function: S 0.
4: for = 2, ..., t do
5:
Generate the watermark code k; = (key, xi-n,i-1). 6: Generate the pseudo-random number r(,)).
7: Update the score function via = 5 + sr((()),, Fi).
8: return S >
Proof. The proof is straightforward. As aligned inverse sampling does not modify the cluster probability, by the property of inverse sampling, we have Pa (z) = Eg Pw (z|0))-
With the aforementioned probability assignments, based on the pseudo-random number and the current token æ.
define statistical score for the detector
Definition 4.2. Given the pseudo-random number and the current token x, the detection score s(r, is defined as
s(r, ((r,))== = {
if : where Ci.
otherwise.
Note, when Pr(ci) < 1.2 aligned inverse sampling may sample tokens from clusters other than even if the pseudo-random number r falls within 1-11 i] empirical results indicate that the overall detectability of aligned inverse sampling still surpasses that of the regular inverse sampling reweight strategy.
This can reduce detection accuracy. However,
4.3 ALIGNED-IS
With the aligned inverse sampling reweight strategy, we can construct our watermarking algorithm ALIGNED-IS. ALIGNED-IS consists of watermark generator and a watermark detector. In the watermark generator, the watermarked token step is generated using secret key key and the prior n-gram content æt-n,t-1 as the watermark code 0,. The pseudo-random number r re(0.) ) is then generated based on 0,. Following the inverse sampling reweight strategy, the cluster C2(9,) is selected based re (0.). The next token 2; is sampled by randomly selecting token within Ci(,,) according to the token probability of the original language model Par ( #14-1). Following Hu et al. (2023), we use a watermark code history hist to ensure the distortion freeness for multiple generation. the current watermark code is in hist, will sample from the original model's distribution instead of the watermarked distribution. The algorithm detailed in Alg. [
During detection, access to the generated audio and the watermark key key is required. The token sequence F1.t is first recovered using the encoder E of the audio generation model. Then, for each i 1,...,t, the watermark code 0 is generated based on the watermark key and the n-gram

---

Table Detectability comparison of watermarking methods on MMW Book Report, MMW Story, and MMW Fake News with SpiritLM. We report true positive rate at 1 % and 0.1 false positive rate and the median p-value.
MMW Book Report
MMW Story
MMW Fake News
T FPR
TPR@FPE FPR
TPR@FP
Method
Median p-value
Median p-value
Median p-value
1%
0.1%
1%
0.1%
1%
0.1%
(GW(1=1.0)
0.75
0.40
0.002
0.66
0.42
0.004
0.63
0.38
0.003
(GW(5=1.5)
0.93
0.76
1.
0.96
0.90
1.2e-05
0.87
0.71
1.3e-04
KGW((#=2.0)
0.97
0.94
2.8c-07
0.97
0.89
S.1e-07
0,92
0.86
2.5e-06
Unigram(6=1.0) ..)
0.09
0.00
0.241
0.06
0.02
0.241
0.08
0.01
0.198
Unigram(6=1.55 (=555)
0.27
0.12
0.043
0.11
0,042
0.37
0.20
0.040
Unigram(8=2.00
0.56
0.29
0.006
0.55
0.30
0.005
0.53
0.28
0.007
1-reweight
0.68
0.41
0.003
0.53
0.006
0.68
0.43
0.002
0.55
0.27
0.44
0.26
0.017
DiPmark(a=U.4)
0.63
0.41
0,002
0.57
0.33
0.004
0.61
0.34
0.003
IT'S
0.77
0.64
2,0c-04
0.82
0.70
2.0c-04
0.83
0.74
2.0c-04
ALIGNED-IS
0.88
0.77
8.0e-05
0.92
0.80
4.6e-05
0.97
0.82
4.9e-05
Table 3: Detectability comparison of watermarking methods Dolly CW, Longform QA, and
Librispeech with SpiritLM. We report true positive rate at 1% and 0.1% false positive rate and the median p-value.
Dolly CW
Longform QA
Librispeech
TPR@ FPR
@PR@FPR
TPR@FPR
Method
Median p-value
Median p-valuc
Median p-valuc
1%
0.1%
1%
0.1%
1%
0.1%
KGW(6=1.0)
0.69
0.44
0,003
0.75
0.44
0.001
0.69
0.42
0.002
KGW(5=1155)
0.93
0.83
2.4c-05
0.95
0.84
1.8c-05
0.97
0.86
1.3e-05
KGW((=2.0)
0.98
0.90
6.0e-07
0.99
0.92
3.4e-07
0.99
0.97
9.7e-08
(0)g)am(6=1.0)
0.10
0.02
0,268
0.15
0.164
0.06
0.01
0.268
Unigram(8=1 (151.5)
0.32
0.16
0.057
0.40
0.17
0.021
0.25
0.07
0.060
Unigram(K=2.0)
0.43
0.23
0.016
0.56
0.32
0.006
0.48
0.23
0.012
7-reweight
0.63
0.43
0.002
0.005
0.71
0.46
0.001
DiPmark(a=0.3)
0.45
0.19
0,019
0.50
0.25
0.010
0.60
0.31
0.006
DiPmark(a=U044
0.58
0.34
0,005
0.53
0.29
0.009
0.69
0.45
0.002
ITS
0.79
0.62
4.0e-04
0.86
0.73
2.0e-04
0.90
0.83
2.0c-04
ALIGNED-IS
0.92
0.80
2.3c-05
0.96
0.82
1.6e-05
0.95
0.84
6.7e-06
content.
Subsequently, the pseudo-random number r, r0,) is recovered based 8,. Next,
calculate the statistical score s(r;,0.), following Definition 4.2 The final statistic is given by S(x).)) = Σi_ (.))
Under the null hypothesis, S(x1.) follows a binomial distribution with success rate of Thus, have the following tail bound derived from the Hoeffding's inequality: Pr(S(#1.) 2 k) < exp(-2t( 2)² By setting threshold on the false positive rate (e.g. FPR=1%), can calculate threshold by solving exp(-21(} 2) 0.01. If the score S(w):) is greater than z, reject the null hypothesis and claim that the sentence is watermarked. The detection algorithm is in Alg.
Experiments
Baselines. We evaluate the performance of our methods against various statistical watermarking baselines, including two biased watermarking approaches, KGW (Kirchenbauer et al. 2023) and
Unigram (Zhao et al. 2023), as well as three unbiased watermarking algorithms, y-reweight (Hu et al. 2023), DiPmark (Wu et al. (023b), and ITS-edit (Kuditipudi et al. (2023). We also compare our in- in-generation method with state-of-the-art post hoc watermarking methods Appendix
Models and Datasets. We evaluate our watermarking approach A ALIGNED-ISON raw audios gener- ated by text-speech aligned conversational models. This allows us for flexibility to prompt speech generation with both text and speech prompts. We use the models SpiritLM (Nguyen et al. 2025)
7

---

Table 4: Detectability comparison of watermarking methods on Dolly CW, Longform QA, and
Finance QA with SpeechGPT. We report true positive rate at I % and 0.1% false positive rate and the median p-value.
Dolly CW
Longform QA
Finance QA
TPR @ FPR
TPR@FFP FPR
TPR@FPR
Method
Median p-value
Median p-value
Median p-value
1%
0.1%
1%
0.1%
1%
0.1%
KGW(6=1.0)
0.32
0.14
0.048
0.40
0.19
0.023
0.36
0.21
0.024
(G
0.56
0.36
0,005
0.72
0.48
0,001
0.68
0.52
8.2e-04
KGW((#=2.0)
0.73
0.51
7.4c-04
0.84
0.69
4.2c-05
0.80
0.70
7.5e-05
Unigram(6=1.0 ..)
0.17
0.06
0.122
0.25
0.10
0.054
0.21
0.07
0.061
Unigram(6=1.5) (=555)
0.40
0.21
0.023
0.55
0.64
0.37
0.003
Unigram(8=200
0.57
0.38
0.004
0.71
0.54
6.1c-04
0.72
0.59
3.0e-04
1-reweight
0.37
0.15
0.047
0.46
0.015
0.56
0.32
0.007
D)Pmark((=0.33)
0.23
0.33
0.17
0.032
DiPmark(a=U.4)
0.29
0.12
0.049
0.46
0.24
0,014
0.54
0.25
0.008
ALIGNED-IS
0.82
0.69
5.6e-05
0.95
1.8e-0
0.94
0.89
1.7e-07
Table 5: Robustness comparison of watermarking methods for Longform QA with SpiritI under signal processing attacks. We report TPR at 1 % FPR. 'Dist.' refers distorted watermarks and 'Dist. free' refers distortion watermarks.
No
Echo
Gauss. noise
Lowpass (40%)
Smooth samp.)
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(0.9)
(1.1)
KGW(8=1.0)
0.75
0.64
0.77
0.77
0.80
0.13
0.11
KGW(8=1.5)
0.95
0.95
0.97
0.96
0.96
0.24
0.27
KGW(8=2.0)
0.99
0.97
0.99
0.99
0.99
0.35
0.35
Dist.
Unigram(8=1.0)
0.15
0.15
0.03
0.16
0.11
0.01
0.02
Unigram(8=115 1.5)
0.40
0.41
0.13
0.39
0.30
0.07
0.05
Unigram(6=2.0)
0.56
0.57
0.30
0.56
0.47
0.13
0.08
7-reweight
0.56
0.56
0.44
0.58
0.56
0.17
0.09
DiP(a=0.3)
0.50
0.50
0.35
0.48
0.49
0.09
0.10
Dist.
DiP(a=0.4)
0.53
0.52
0.39
0.54
0.49
0.14
0.10
free
ITS
0.86
0.85
0.79
0.88
0.85
0.06
0.00
ALIGNED-IS
0.96
0.93
0.93
0.94
0.94
0.31
0.28
and SpeechGPT (Zhang al. 2023 for the speech generation tasks. For text prompting, follow
Kirchenbauer et al. (2023): | Hu et al. (2023) and include three MMW datasets (Piet et al. (023).. Dolly CW (Conover al. 2023), and two tasks from WaterBench (Tu et al. 2023). For speech prompting.
we use the validation set of LibriSpeech (Panayotov et al. 2015).
Watermarking parameters. We evaluate the detectability of ALIGNED-LE IS on the speech gener- ation task with different audio generation models. We generate 500 examples for each task. We use the prefix 1 1-gram together with a secret key as the watermark keys. We select a e (0.3, 0.4) for DiPmark, and 5 (1.0, 1.5, 2.0} and 7 0.5 for KGW watermark (Kirchenbauer al. 2023), E (1.0, 1.5, 2.0} for Unigram (Zhao et af.) 2023). For ALIGNED-IS, we partition the token- embedding space into 20 clusters using the k-mmans algorithm, then perform linear assignment to ensure that the resulting centroids are sufficiently separated to accommodate potential retokeniza- tion errors. We justify the choice of h 20 Appendix E.] All experiments are conducted NVIDIA A6000 GPU.
5.1 Detectability
Following the evaluation metric of the previous works (Kirchenbauer et al. 2023 Wu et al. 2023b
report the true positive rate guaranteed false positive rates, i.e., TPR @ = 1%, 0.1% Notice, as the detectors of eeS- do not provide a theoretical guarantee, we report the true positive rate the estimated false positive rate following their detecting algorithms.
From Table 2 3 and we see that ALIGNED-IS achieved the best detectability comparing with all other unbiased watermarks, at least 10% improvement on all TPR @ FPR metrics. Besides, ALIGNED-

---

Table Robustness comparison of watermarking methods for Longform QA with SpiritLM under codec-based, quantizing, and denoising attacks. We report TPR at 1% FPR. 'Dist.' refers distorted watermarks and 'Dist. free' refers distortion-fres watermarks.
EnCodec (24kHz)
MP3
MP3
Opus (16kbps)
Opus (31kbps)
Quant. (64-bit)
Watermark
Denoise
(32kbps)
(40kbps)
KGW(6=1.0)
0.64
0.51
0.33
0.65
0.74
0.68
0.65
KGW(d=1.55
0.93
0.88
0.86
0.92
0.95
0.92
0.93
KGW(d=2.0)
0.98
0.94
0.93
0.97
0.98
0.97
0.97
Dist.
Uni(ő=1.0)
0.06
0.12
0.12
0.12
0.14
0.00
0.06
Uni(Ø=1.55
0.26
0.32
0.34
0.33
0.37
0.10
0.22
Uni(6=2.0)
0.41
0.53
0.49
0.49
0.53
0.20
0.38
7-reweight
0,51
0.42
0.46
0,54
0,56
0.57
0.52
DiP(a=0.3)
0.46
0.30
0.34
0,46
0.48
0,60
0.44
Dist.
DiP(4)0.4)
0.49
0.41
0.41
0.47
0.51
0.56
0.48
free
ITS
0.77
0.74
0.78
0.80
0.88
0.63
0.71
ALIGNED-IS
0.92
0.80
0.84
0.91
0.92
0.88
0.92
IS outperformed the biased watermarking algorithm KGW and Unigram most cases, and achieved comparable performance with the strongly biased KGW(0=2.0).
Time Efficiency. Similar to KGW, Unigram, and DiPmark watermarking approaches, the minimal computational overhead introduced by the ALIGNED-IS generator occurs solely during the adjustment of token probabilities in the generation stage, and can be elegantly impelemented a logits processor in popular deep learning framweorks. Moreover, the ALIGNED-IS detector is model-agnostic.
5.2 Robustness
Audio in the wild is subject to various modifications. Hosting platforms use codecs for efficiency, and users employ editing software either for recreational purposes, or specifically to erase water- marking signals. To evaluate the robustness of ALIGNED-IS under realistic channel conditions, we apply diverse suite thirteen single-channel audio attacks. We follow the no-box attacks of BudicMarkBench (Liu al. 2024b), which includes common signal porcessing modifications (time dilation, echo, Gaussian noise addition, lowpass filtering, temporal smoothing, quantization), popular audio codings (Opus, EnCodec (Détossez et al.) 2023). ) and MP3), well as the denoising attack of López-López et al. (2024). The detailed settings are in Section
The results are summarized in Band Table We report the true positive rate at fixed false positive rate of 1% for each watermark across range of attack types and strengths. ALIGNED- IS consistently exhibits the strongest robustness, outperforming all distortion-free watermarking baselines in reliably detecting watermarked audio under adversarial conditions. Experiments on two additional datasets per model are included in F.2
5.3
Audio quality
We employ non-intrusive speech quality metrics validate unbiasedness, due to the inherent stochas- ticity of token-based statistical watermarking methods. Specifically, we utilize the Fréchet Audio Distance (FAD) (Kilgour al. 2019), metric designed directly quantify the divergence between two distributions. We compute the FAD scores between the watermarked audios and their unwater- marked counterparts generated from identical prompts and the unwatermarked model. As baseline (no watermarking scenario), we generate two distinct sets of audio samples from the unwatermarked model, using the same prompts but different random seeds, and calculate the FAD scores between these two sets. Figure 2 shows that ALIGNED-IS and the other distortion-fiee free watermarking schemes preserve audio quality on par with the unwatermarked baseline, whereas distortion- based approaches such KGW and Unigram noticeably degrade it.

---

FAD scores
0.05
0.00
watermark
KGW(5=1.5) KGW(5=1.0)
KGW(5=2.0)
Unigram(6=1100
Unigram(6=115)
DiPmark(a=003) Unigram(5=2.0)
DiPmark(a==0 4)
gamma- remna-reweight
Alignad-IS
FAD scores
0.02
0.00
No watermark
KGW(5=1.0)
KGW(5=1.5)
KGW(6=2.0)
Unigram(6=110 (6=10)
Unigram(6=115)
Unigram(6=2.00)
..5)
DiPmark(a-o.3)
DiPmark(a=-..) .)
gamma- reweight
Alignad-IS
Figure 2: Audio-quality impact of watermarking methods. Top: FAD scores on the Dolly CW dataset evaluated with SpiritLM. Bottom: FAD scores on the LibriSpeech dataset evaluated with SpiritLM.
6 Conclusion
In conclusion, we propose ALIGNED-IS, novel distortion-free watermarking framework tailored specifically for autoregressive audio generation models. Leveraging aligned inverse sampling. ALIGNED-IS ensures traceability and accountability in synthetic audio outputs without any degra- dation in audio quality. Comprehensive empirical evaluations across diverse datasets and audio generation architectures demonstrate the efficacy and robustness of our approach. Our method thus represents meaningful advancement watermarking technology, enhancing the security and integrity of synthetic audio and supporting trustworthy digital communication.
References
Aaronson, S. My Al safety lecture for UT effective altruism,, 2022. URL SRLpttt:///scotaoonooo
blog/?p=688 6823 6823
An, L., Liu, Y., Liu, Y., Zhang, Y., Bu, Y., and Chang. S. Defending Ilm watermarking against spoofing attacks with contrastive representation learning, arXiv preprint vrXiv:2504.06575. 2025.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-12444 12460, 2020.
Borsos, 7., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. Audiolm: language modeling approach to audio generation. arXiv preprint arXiv:2209003143 2022.
Chen, G., Wu, Y., Liu, S., Liu, Du, X., and Wei, Wavmark: Watermarking for audio generation. arXiv preprint arXiv:2308. /2770, 2023.
Chen, R., Wu, Y., Chen, Y., Liu, C., Guo, J., and Huang, H. A watermark for order-agnostic language models. In The Thirteenth International Conference on Learning Representations, 2024a.
Chen, R., Wu, Y., Guo, J., and Huang, H. Improved unbiased watermark for large language models. arXiv preprint arXiv:2502. 2025a.
Chen, Wu, Y.. Guo, J., and Huang, H. De-mark: Watermark removal in large language models. In Forty-second International Conference on Machine Learning, 2025b.
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et Wavim: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1666):505-1118. 2022.
10

---

Chen, S., Liu, S., Zhou, L., Liu, Y.. Tan, X., Li, Zhao, S., Qian, Y., and Wei, F. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.055370, 2024b.
Christ, M., Gunn, S., and Zamir, O. Undetectable watermarks for language models. arXiv preprini arXiv:2306.09194, 2023.
Conover, M., Hayes, M. Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wen- dell, P.. Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction-tune tunde 1lm. 2023. URL https: / /www. databricks com haamm/log//a bttp//ltggg///e/e/////////// www.raable-nssstttitooooo instruction-tunett
Cumlin, F., Liang, X., Ungureanu, V., Reddy, C. K., Schüldr, C., and Chatterjee, S. Dnsmos pro: A reduced-size dnn for probabilistic mos of speech. In 25th Interspeech Conferece 2024, Kos Island, Greece, Sep I 2024-Sep 5 2024, pp. 4818 4822. International Speech Communication Association, 2024.
Dathathri, S., See, A., Ghaisas, S., Huang, P.-S., McAdam, R., Welbl, J., Bachani, V., Kaskasoli, A., Stanforth, R., Matejovicova, T., et al. Scalable watermarking for identifying large language model outputs. Nature, (14(80355)8188-22, 2024.
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transac- tions on Machine Learning Research, 2023.
Feng, X., Zhang, H., Zhang, Y., Zhang, L. Y., and Pan, S. Bimark: Unbiased multilayer watermarking for large language models. arXiv preprint arXiv2506..1662, 2025.
Fu, C.,, Lin, H., Long, Z., Shen, Y., Zhao, M., Zhang, Y., Wang, X., Yin, D., Ma, L., Zheng, X., He, R., Ji, R., Wu, Y., Shan, C., and Sun, Vita: Towards open-source interactive omni multimodal 2024.
Ilm. arXiv preprint urXiv:2408.0555
Ge, Y., Zhao, S., Zeng, Z., Ge, Y., Li, C., Wang, X., and Shan, Y. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:3310.01218, 2023.
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Iransactions on audio, speech, and language processing, 29:3451-3486 2021.
Hu, Y., Liu, Y., Lv, S., Xing, M., Zhang, S., Fu, Y., Wu, J., Zhang, B., and Xie, L. Decrn: Deep complex convolution recurrent network for phase-aware speech enhancement, Interspeech 2020, 2020.
Hu, Z., Chen, L., Wu, X., Wu, Y., Zhang, H., and Huang, H. Unbiased watermark for large language models. arXiv preprint arXiv:2310. 10669, 2023.
Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. Interspeech, 2019.
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. watermark for large language models. arXiv preprint arXiv:2301. 10226, 2023.
Kong. J.. Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022-1170000 17033, 2020.
Kuditipudi, R., Thickstun, J., Hashimoto, T., and Liang, P. Robust distortion free watermarks for language models. arXiv preprint vrXiv:2307.15593 2023.
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte. B., Nguyen. T.-A., Copet, J., Baevski, A., Mohamed, A., et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021.
Lie, W.-N. and Chang, L.-C. Robust and high-quality time-domain audio watermarking based on low-freee cow-frequency amplitude modification. IEEE transactions on multimedia, 8(1):46-59, 2006.
11

---

Liu, C., Zhang, J., Zhang, T., Yang, X., Zhang, W., and Yu, N. Detecting voice cloning attacks via timbre watermarking. Nenwork and Distributed System Security Symposium, 2024a. doi: 10.14722hmss.2224422200..
Liu, H., Guo, M., Jiang, Z., Wang, L., and Gong, N. Audiomarkbench: Benchmarking robustness of audio watermarking. Advances in Neural Information Processing Systems, 37:52241-52248 2024b.
Liu, Y. and Bu, Y. Adaptive text watermark for large language models. In Proceedings of the 41st International Conference Machine Learning, pp- 20718-30737, 2024.
Liu, Y., Song. Y., Ci, H., Zhang. Y., Wang, H., Shou, M. Z.. and Bu, Y. Image watermarks are removable using controllable regeneration from clean noise. In The Thirleenth International Conference on Learning Representations, 2025a.
Liu, Y., Zhao, X., Song, D., and Bu, Y. Dataset protection via watermarked canaries in retrieval- augmented Ilms. arXiv preprint arXiv:2502.10673, 2025b.
López-López, Á., Rosello, E., and Gomez, A. M. Speech watermarking removal by dnn-based speech enhancement attacks. In Proc. IberSPEECH 2024, pp. 16-20, 2024.
Lu, J., Clark, C., Lee. S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26439-26455, 2024.
Mittag, G., Naderi, B., Chehadi, A., and Möller, S. Nisqa: A deep cnn-selffattentionn model for multidimensional speech quality prediction with crowdsourced datasets. In Pruc. Interspeech 2021, pp. 2127-2131, 2021.
Nguyen, T. A., Muller, B., Yu, B., Costa-Jussa, M. R., Elbayad, M., Popuri, S., Ropers, C., Duquenne, P.-A., Algayres, R., Mavlyutov, R., et al. Spirit-Im: Interleaved spoken and written language model. Transactions of the Association for Computational Linguistics, 13:30-52 52, 2025.
O' O'Reilly, P., Jin, 7., Su, J., and Pardo, B. Deep audio watermarks are shallow: Limitations of post-hoc watermarking techniques for speech. In The 1st Workshop on GenAI Watermarking, 2025.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: ast corpus based on public domain audio books. In 2015 IEEE international conference on acousties, speech and signal processing (ICASSP). pp. 5206-5210. IEEE, 2015.
Piet, J., Sitawarin, C., Fang. V., Mu, N., and Wagner, D. Mark my words: Analyzing and evaluating language model watermarks. arXiv preprint arXiv:2312.00273, 2023.
Rubenstein, K., Asawaroengchai, C., Nguyen, D., Bapna, A., Borsos, Z.,, Quitry, C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. 12925, 2023.
San Roman, R., Fernandez, P., Elsahar, H., Défossez, A., Furon, T., and Tran, Proactive detection of voice cloning with localized watermarking. In International Conference on Machine Learning. volume 235, 2024.
Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/P16-11
URL https
//aclanthology- . www.oog..-........... 1162/
Tang, Z., Yang, Z., Khademi, M.. Liu, Y., Zhu, C., and Bansal, M. Codi-2: In-context interleaved and interactive any-to-any generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27425-27434, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models, arXiv preprint ar arXiv:2307.0022000 2023.
12

---

Tu, S., Sun, Y., Bai, Y., Yu, J., Hou, L.. and Li, J. Waterbench: Towards holistic evaluation of watermarks for large language models. arXiv preprint arXiv:2311.07138. 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, pp. 1000-6010, 2017.
Wu, S., Liu, J., Huang, Y., Guan, H., and Zhang, S. Adversarial audio watermarking: Embedding watermark into deep feature. In 2023 IEEE International Conference Multimedia and Expo (ICME), pp. 61-66. IEEE, 2023a.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. Next-gpt: Any-to-any multimodal Ilm. In Proceedings of the International Conference on Machine Learning, pp. 53366-53397, 2024.
Wu, Y., Hu, Z., Zhang, H., and Huang, H. Dipmark: stealthy, efficient and resilient watermark for large language models. arXiv preprint arXiv:2310.07711 10, 2023b.
Wu, Y., Chen, R., Milis, G., Guo, J., and Huang, H. A watermark for auto-regressive speech generation models. Proc. Interspeech 2025, pp. 3474-3478, 2025.
Zhan, J., Dai, J., Ye, J., Zhou, Y., Zhang, D., Liu, Z., Zhang, X., Yuan, R., Zhang, G., Li, L., et al. Anygpt: Unified multimodal I1m with discrete sequence modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9637-9662, 2024.
Zhang. D., Li, S., Zhang, X., Zhan, J., Wang. P., Zhou, Y., and Qiu, Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15757-15773, 2023.
Zhang, J., Chen, R., Yang, Y., Mai, P., Huang, H., and Pang. Y. Leave no trace: Black-box
detection of copyrighted dataset usage in large language models via watermarking. arXiv preprint arXiv:2510.02962, 2025.
Zhao, X., Ananth, P., Li, L., and Wang. Y.-.. Provable robust watermarking for ai-generated text. arXiv preprint 200i::2306./43333 2023.
13

---

A Limitations
ALIGNED-IS method relies on the assumption that the retokenization mismatch adequately captured by clustering. However, there is no formal guarantee that the clustering process fully captures all forms retokenization errors, especially when new audio patterns or novel speech artifacts are introduced. Besides, for each model, need to perform the clustering of tokens, which introduces an additional computational step.
B Missing Algorithms
Algorithm 3 Aligned inverse sampling.
1: Input: Cluster probabilities Pr(ct)...,PPa Pr(ce), sorted from max to min probabilities. Watermark
code 0.
2: Initialize an overlapped dict to store the overlapped regions.
Initialize cluster list and prob list for inverse sampling.
4: # Rearrange probabilities of clusters within [0.
5: for -,....,h do
6: cluster litp__isttppppd(cc
7:
if Pr(c,) 2 1/h then
8:
prob_ list list.append(1/h))
9:
Store the overlapped regions
10:
Pr(c)) - 1/hb)
11:
else
12:
prob list...pppend(p((c))
13:
diff=1/h - Pr(c,)
14:
# Use the overlapped regions fill the empty region
15:
while diff do
16:
for j Eoverlapped -icpteedddddddodeeeeeee dict do
17:
cluster (luster_lsttappendd() hoob_llistappendmmnn((dee
18:
overlapped
19:
hii-adiffooerrlapee_ddilll
20:
overlapped_ overlapped_diccli)=mmxxx overlapped diff)
21: # Sampling the rearranged interval.
22: Pseudo-randont sampling r()) e(0, 1] seeded by 0.
23: Find s.t. r(0) E[sum(pro)) list[O: i)), sum(prob list(O: i+1])
24: Randomly sample the token (following their original probability) from [tuster]list[]]
25: return x
C Comparison with post hoc methods
We compare our in-generation method with the state-of-the-art post hoc watermarking methods AudioSeal (San Roman al., (2024) and WavMark (Chen al. 2023). Our findings indicate that, established in existing literature, post hoc watermarking methods are neither robust, nor distortion-free. Statistical watermarking is promising solution for embedding zero-bit watermarks to distinguish artificially generated audio. Specifically, our audio-aware method achieves new state-of-the--ad robustness and distortion-frrenesss.
C.1 Detectability & Robustness
First, we verify the results of O'Reilly et al. (2025), who identified that post hoc methods are robust to few attacks, but extremely vulnerable to others, making them unpractical in real-world applications. Token-based statistical watermarking more robust to wide variety of attacks, since the watermarking information is embedded into the generated audio itself, in cotrast to overlaying mid-frequency content on the audio. We present our results in Tables through TO for SpiritI.M, and Tables through for SpeechGPT. Notice that post hoc methods mostly robust to low-frequency
14

---

Table 7: Robustness comparison post hoc watermarking methods for Longform QA with SpiritLM
under signal processing attacks.
No
Echo
Gauss. noise
Lowpass (40%)
Smooth
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(6 samp.)
(0.9)
(1.1)
AudioSeal
1.00
0.88
0.07
1.00
1.00
0.00
0,00
WavMark
1.00
1.00
1.00
1.00
1.00
1.00
1.00
ALIGNED-IS
0.96
0.93
0.93
0.94
0.94
0.31
0.28
Table 8: Robustness comparison post hoc watermarking methods for Longform QA with SpiritLM under codec-basee and quantizing attacks.
EnCodec
MP3
MP3
Opus (16kbps)
Opus (3 (31kbps)
Quant.
Watermark
(24kHz)
(32kbps)
(40kbps)
(64-bit)
AudioScal
0.00
0.18
0.81
0.00
0.90
0.00
WavMark
0.00
1.00
1.00
0.96
1.00
0.06
ALIGNED-IS
0.92
0.80
0.84
0.91
0.92
0.88
perturbations (since they alter inaudible frequencies) and higher- higher-bitrate bitrate codings, but are very fragile otherwise.
Detectability is reported as the TPR at 1% FPR for ALIGNED-IS. however the post hoc methods do offer theoretical detectability guarantee. For AudioSeal, the detection result is the probability of the audio being watermarked, so we report that probability thresholded at 99%. For WavMark 16-bit payload is embedded into the audio, and detection returns either an empty, or decoded payload. We consider successful detection when a payload is not empty, even if it has non-zero bit error rate.
C.2 Audio quality
We then showcase that post hoc methods are inevitably harmful to audio quality, since operating on the waveform level significantly deteriorates the FAD score. Notably, audios processed by WavMark have audible high-free sigh-freueency artifacts, which justifies its higher detectability. We present the audio quality in Tables 15 through [8] The mean opinion score (MOS) is also reported using the estimators NISQA and DNSMOSPro, from Mittag et al. (2021) and Cumlin et al. (2024).
D Experimental settings
D.1 Attack Suite
To evaluate the robustness of ALIGNED-IS under realistic channel conditions, apply suite of thirteen single-channel audio attacks. Each attack is tuned to a moderate, perceptually acceptable strength.
Echo (0.05 s) Echo with a 50 ms delay.
Table 9: Robustness comparison of post hoc watermarking methods for LibriSpeech with SpiritLM under signal processing attacks.
No
Echo
Gauss, noise
Lowpass (40%)
Smooth
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(6 samp.)
(0.9)
(1.1)
AudioSeal
1.0
0.82
0.05
1.00
1.00
0.00
0.00
WavMark
1.00
1.00
1.00
1.00
1.00
1.00
1.00
ALIGNED-IS
0.95
0.92
0.90
0.93
0.92
0.28
0.21
15

---

Table 10: Robustness comparison of post hoc watermarking methods for LibriSpeech with Spiritl M under codec-based and quantizing attacks.
EnCodec (24kHz)
MP3
MP3
Opus (16kbps)
Opus (31kbps)
Quant.
Watermark
(32kbps)
(40kbps)
(64-bit)
AudioSeal
0.00
0.18
0.84
0.00
0.89
0.00
WavMark
0.00
1.00
1.00
0.99
1.00
0.06
ALIGNED-IS
0.90
0.75
0.80
0.89
0.91
0.85
Table 11: Robustness comparison of post hoc watermarking methods for Longform QA with SpeechGPT under signal processing attacks.
No
Echo
Gauss, noise
Lowpass
Smooth
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(40%)
(6 samp.)
(0.9)
(1.1)
AudioSeal
1.00
0.77
0.00
1.00
1.00
0.00
0.00
WavMark
1.00
1.00
1.00
1.00
1.00
1.00
1.00
ALIGNED-IS
0.95
0.93
0.93
0.95
0.92
0.38
0.22
Table 12: Robustness comparison of post hoc watermarking methods for Longform QA with
SpeechGPT under codec-based and quantizing attacks.
EnCodec
MP3
MP3
Opus
Opus (3 (3bbp))
Quant. (64-bit)
Watermark
(24kHz)
(32kbps)
(40kbps)
(16kbps)
AudioSeal
0.00
0.02
0.05
0.00
0.26
0.00
WavMark
0.00
1.00
1.00
0.74
1.00
0.30
ALIGNED-IS
0.91
0.82
0.84
0.95
0.96
0.85
Table 13: Robustness comparison of post hoc watermarking methods for Finance QA with SpeechGPT
under signal processing attacks.
No
Echo
Gauss. noise
Lowpass
Smooth
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(40%)
(6 samp.)
(0.9)
(1.1)
AudioSeal
1.00
0.84
0.01
1.00
1.00
0.00
0.00
WavMark
1.00
1.00
0.99
1.00
1.00
1.00
1.00
ALIGNED-IS
0.94
0.94
0.94
0.94
0.93
0.38
0.24
Table 14: Robustness comparison of post hoc watermarking methods for Finance QA with SpeechGPT under codec-based, quantizing, and denoising attacks.
EnCodec (24kHz)
MP3
MP3
Opus
Opus (31kbps)
Quant. (64-bit)
Watermark
(32kbps)
(40kbps)
(16kbps)
AudioSeal
0.00
0.01
0.06
0.00
0.28
0.01
WavMark
0.00
1.00
1.00
0.74
1.00
0.29
ALIGNED-IS
0.91
0.89
0.88
0.95
0.94
0.87
Table 15: Quality comparison with post hoc methods for Dolly CW with SpiritLM
MOS 1
Method
FAD
NISQA
DNSMOSPro
No watermark
0.0487
3.915
3.713
AudioSeal
0.3083
3,813
3.779
WavMark
1.8233
4.167
4.215
ALIGNED-IS
0.0512
3.860
3.763
16

---

Table 16: Quality comparison with post hoc methods for Longform QA with SpiritLM
MOS 1
Method
FAD +
NISQA
DNSMOSPrO
No watermark
0.0337
3.934
3.766
AudioSeal
0.3061
3.830
3.811
WavMark
1.7910
4.160
4.190
ALIGNED-IS
0.0416
3.958
3.764
Table 17: Quality comparison with post hoc methods for Finance QA with SpiritLM
MOS 1
Method
FAD +
NISQA
DNSMOSPrO
No watermark
0.0233
3.964
3.768
AudioSeal
0.3081
3.901
3.777
WavMark
1.7228
4.212
4.158
ALIGNED-IS
0.0515
3.948
3.766
Gaussian noise (30 dB SNR) Injects additive white Gaussian noise to achieve an output signal-to-noise ratio of 30 dB.
Low-pass filter (40 % of Nyquist) Applies low-pass filter with cut-off frequency equal 40% of the Nyquist rate.
Smoothing (6-sample moving average) Applies moving aveerae filter of width samples.
Speed perturbation (0.9x and 1.1x) Interpolates the waveform to speed up or slow down, accordingly.
EnCodec (24 kHz) Re-encodes the audio with Meta's EnCodec neural codec (ééossez et al. 2023) at 24 kHz bandwidth.
MP3 recompression (32 kbit/s and 40 kbit/s) Re-encodes the waveform using at the given constant bit rate.
Opus recompression (16 kbit/s and 31 kbit/s) Re-encodes the waveform at the given constant bit rate.
Quantization (64 levels) Uniformly quantizes samples to 64 discrete amplitude levels.
Denoising Applies the DCCRN (Hu et al. 2020) denoising network to the waveforms perturbed by the Gaussian noise 30 dB.
Each attack is applied independently to the model outputs and no attack stacking is used.
Table 18: Quality comparison with post hoc methods for LibriSpeech with SpiritLM
MOS 1
Method
FAD +
NISQA
DNSMOSPro
No watermark
0.0054
3.966
3.780
AudioSeal
0.2918
3.915
3.798
WavMark
1.6509
4.243
4.203
ALIGNED-IS
0.0065
3.959
3.798

---

Table 19; Ablation on the optimal number of clusters for SpiritLM.
h
TPR @FR=1%
Median p-value
10
0.798343
0.000473744
20
0.922018
2.27798e-05
30
0.724234
0.00165162
40
0.904494
6.96696e-05
80
0.696884
0.00124669
100
0.858696
0.000264207
120
0.676093
0.00145675
150
0.687927
0.002948
Table 20: Ablation on the optimal number of clusters for SpeechGPT.
h
TPR PR@FFP=11%
Median p-value
10
0.666667
0.000672034
20
0.816092
5.56062e-05
30
0.660377
0.00143836
40
0.548023
0.00592188 88
80
0.754144
0,000392686
100
0.248538
0,0748774
120
0,507553
0.0092863
150
0.784703
0.000100627
E
Additional Ablations.
E.1 Number of clusters
We conducted an ablation study on the number of clusters for both models using the Dolly CW dataset, with generation settings identical to the main experiments. We resent the results in Tables 19 and 20, which show that detectability initially increases with more clusters but begins to decline beyond certain point. This trend arises because SpiritI.M has approximately 500 audio tokens, and using too many clusters Ieads to overly fine partitions that fail to effectively mitigate retokenization errors. We find that strikes a good balance and yields optimal detectability, supporting our choice in the main experiments. For the non-clustering bascline, we assign tokens randomly to h=20 clusters and apply ALIGNED-IS.
E.2 Dependence on generation length
The length of audio token sequences used for detection is typically around 500, which corresponds to approximately 10-20 seconds of audio. The duration is affected by the tokenizer and vocoder frame rates. Some variation arises because the duration of each token may differ depending on the model's duration prediction. In Table 2T, we present detection results randomly cropped segments of watermarked audio with varying durations. As shown, detection performance improves with longer audio segments, achieving reliable detectability for audio durations exceeding seconds.
E.3 Robustness to time shift
Due to the reliance on accurate audio retokenization which assumes frame alignment, the proposed watermark is not inherently robust misalignment attacks, like speed modifications, as established in Tables 55 andd However, cropping attacks or time shifts can deceive detection only if they operate on small offsets (smaller than the frame size). We performed an experiment with the Longform QA dataset and the SpiritLM model, whose frame size is 645 samples. We uniformly sampled 8 time shifts, with shifts larger than 322 being equivalent to negative shifts of less than 50% of frame length. In Table 22 we observe our watermark's strength decrease and then increase, but still maintaining high detectability at small offsets. An effective practical defense against cropping attacks would be to simply run detection on few slightly misaligned versions of the subject audio.
18

---

Table 21: The impact of audio length on detection, for the Longform QA dataset with SpeechGPT.
Time (sec)
@PR@FPR=1%
Median p-value
22.
0.5000
1.0306e-02
330
0,6707
1.7270e-03
4.0
0.8232
4.8318e-04
5,0
0.8659
7.5983e-05
660
0.9207
2.1367e-05
770
0.9329
8.7637e-06
8.0
0.9390
4.2917e-06
900
0.9451
2.0837e-06
Table 22: The impact of audio length detection, for the Longform QA dataset with SpeechGPT.
offset (samples)
TPR @ 1%R=1%
Median p-value
0
0.94
2.8014c-05
80
0.92
1.2973e-05
160
0.76
0.0001
240
0.58
0.0041
320
0.43
0.0191
400
0.37
0.0318
480
0.54
0.0069
560
0.88
0.0001
640
0.94
2.6169e-05
E,4 Quality comparison
We evaluate speaker similarity using WavLMMBase (Chen al. 2022), and ASR-CER/WER using HuBERT-large fine-tuned model (Hsu et al. 2021). with watermarked audios. We use SpiritLM with Librispeech dataset. From the results in Table 23 we observe that distortion-free watermark- ing methods, e.g., Aligned-IS, DiPmark, and 7-reweight, achieve performance comparable to the unwatermarked baseline across all metrics. In contrast, biased and post-hoc watermarking methods noticeably degrade the generation quality.
Supplementary Experimental results.
F1 Detectability
We present an additional watermark detectability evaluation on the Finance QA dataset with SpiritLM in Table 24 From Table 24 we see that ALIGNED-IS IIS achieved the best detectability comparing with other unbiased watermarks, at least 10% improvement on all TPR @TPR metries. Besides,
Table 23:
Comparison of watermarking methods speech quality and recognition metrics. Higher Speaker Similarity and lower ASR-CER/WER indicate better performance.
Method
Speaker Similarity
ASR-CER +
ASR-WER
Baseline
0.7550
0.1039
0.1475
Audioseal
0.7479
0,1086
0.1648
Wavmark
0.7311
0.1143
0.1819
Aligned-IS
0,7548
0.1038
0.1472
DiPmark(0.44
0.7543
0.1045
0.1468
y-reweight
0.7546
0.1043
0.1477
KGW(1.5)
0.7446
0.1093
0.1617
Unigram(1.55
0.7431
0.1102
0.1656
19

---

Table 24: Detectability for Finance QA with SpiritLM
TPR @
Method
Median p-value
1%
0.1%
KGW(d=1.0)
0.71
0.49
0.001
kGW(d=1.5)
0.94
0.79
1.1e-05
KGW(d=2.0)
0.97
0.92
3.2e-07
Unigram(O=1..0)
0.19
0.06
0.151
Unigram(O=11)
0.32
0.15
0.039
Unigram(O=2.0)
0.66
0.39
0.002
Da
0.35
0.18
0.027
DiPmark(a=0.4)
0.48
0.23
0,012
7-reweight
0.56
0.34
0.005
ITS
0.85
0.76
2.0e-04
ALIGNED-IS
0.94
0.79
2.4e-05
Table 25: Robustness comparison of watermarking methods for Finance QA with SpiritLM under
signal processing attacks. We report TPR 1% FPR.
No
Echo
Gauss. noise
Lowpass (40%)
Smooth (6 samp..)
Speed (0.9)
Speed
Watermark
allack
(0.05xec)
(30dB)
(1.1)
KGW()=1.0)
0.71
0.70
0.79
0.74
0.74
0.12
0.14
KGW((==1.5)
0.94
0.94
0.95
0.94
0.94
0.27
0.25
KGW(#=2.0)
0.97
0.98
0.98
0.97
0.98
0.37
0.36
Dist.
Uni(6=1.0)
0.19
0.16
0.04
0.20
0.13
0.03
0.01
Uni(f=1.55
0.32
0.32
0.14
0.30
0.22
0.06
0.05
Uni(6=2.0)
0.66
0.62
0.39
0.66
0.57
0.17
0.12
y-reweighh
0.56
0.52
0.38
0.53
0.58
0.07
0,07
DiP(a=0.3)
0.35
0.38
0.26
0.40
0.38
0.06
0.05
Dist.
DiP(a=0.4)
0.48
0.45
0.34
0.47
0.43
0.10
0.08
free
ALIGNED-IS
0.94
0.93
0.90
0.96
0.93
0.39
0.30
ALIGNED-IS outperformed the biased watermarking algorithm KGW and Unigram in most cases, and achieved comparable performance with the strongly biased KGW(f=2.0).
F22 Robustness
We supplement the robustness evaluation with Tables 25 through 28 for SpiritLM, and Tables 29 through [34] for SPeechGPT. ALIGNED-IS consistently exhibits the strongest robustness, outper- forming all distortion-free watermarking baselines in reliably detecting watermarked audio under adversarial conditions.
G Broader Impact
We introduce ALIGNED-IS, distortion-free watermarking framework for autoregressive audio generation models, addressing the retokenization mismatch that limits traditional methods. Beyond the specific technical udvancement, its broader impact lies in enhancing the security and trustworthiness of Al-generated audio, which is increasingly critical as synthetic media proliferates. Watermarking technologies such as this not only help identify Al-generated content but also have wider applications. including copyright protection (Liu et al. 2025b) Zhang et al.) 2025 Chen et al. 2024a). content authenticity verification, and digital rights management. However, the paper also highlights that robustness remains key challenge (An et al.) 2025 Liu et al. 2025a Chen et al. (025b). Watermarks can be degraded or removed through transformations or noise, emphasizing the ongoing need for more resilient and standardized approaches across modalities
20

---

Table 26: Robustness comparison of watermarking methods for Finance QA with SpiritLM under bodec-based, quantizing, and denoising attacks. We report TPR at 1 % FPR.
EnCodec (24kHzz
MP3
MP3
Opus (16kbps)
Opus (31kbps)
Quant. (64-bit)
Watermark
Denoise
(32kbps)
(40kbps)
KGW(d=1.0)
0.69
0.54
0.51
0.69
0.73
0.64
0.67
KGW(6=1.5)
0.93
0.79
0.81
0.88
0.94
0.90
0.91
KGW(d==.0)
0.99
0.91
0.92
0.94
0.98
0.95
0.97
Dist.
Uni(ð=1.0)
0.10
0.14
0.14
0.12
0.17
0.01
0.08
Uni(ð=1.5)
0,20
0.30
0.30
0.24
0.33
0.05
0.19
Uni(d=2.0)
0.52
0.64
0.61
0.54
0.63
0.18
0.48
7-reweight
0.48
0.36
0.38
0,47
0.55
0.56
0.53
DiP(a=0.3)
0.32
0.27
0.27
0.34
0.37
0.41
0,36
Dist.
DiP(a=004)
0.43
0.35
0.38
0.50
0.42
0.54
0.49
free
ALIGNED-IS
0.91
0.81
0.80
0.92
0.93
0.85
0.89
Table 27: Robustness comparison of watermarking methods for LibriSpeech with SpiritL. under
signal processing attacks. We report TPR 1% FPR.
No
Echo
Gauss, noise
Lowpass (40%)
Smooth (6 samp.)
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(0.9)
(1.1)
KGW()=1.0)
0.69
0.52
0.64
0.55
0.58
0.05
0.09
KGW($=1.5)
0.97
0.86
0.92
0.90
0.91
0.17
0.18
KGW(S=2.0)
0.99
0.97
0.97
0.97
0.97
0.24
0.26
Dist.
Uni(f=1.0)
0.06
0.05
0.01
0.05
0.04
0.01
0.00
Uni(f=1.5)
0.25
0.18
0.07
0.17
0.14
0.02
0.01
Uni(A=2.0)
0.48
0.39
0.18
0.38
0.32
0.04
0.03
y-rewcight
0.71
0.66
0.57
0.66
0.71
0.21
0.13
DiP(a=0.3)
0.60
0.57
0,40
0.56
0.61
0.17
0.13
Dist.
DiP(a=0.4)
0.69
0.66
0.54
0.69
0.71
0.18
0.15
free
ALIGNED-IS
0.95
0.92
0.90
0.93
0.92
0.28
0.21
Table 28: Robustness comparison of watermarking methods for LibriSpeech with SpiritLM under codec-based, quantizing, and denoising attacks. We report TPR at FPR.
EnCodec
MP3
MP3
Opus (16kbps)
Opus (31kbps)
Quant.
Watermark
Denoise
(24kHz)
(32kbps)
(40kbps)
(64-bit)
KGW($=1.0)
0.45
0.29
0.31
0.51
0.53
0.47
0.53
KGW(8=1.5)
0.84
0.64
0.61
0.82
0.90
0.78
0.85
KGW(8=2.0)
0.95
0.84
0.83
0.95
0.97
0.93
0.96
Dist.
Uni(5=1.0)
0.03
0.05
0.04
0.03
0.04
0.00
0.02
Uni(5=1.5)
0.11
0.15
0.14
0.16
0.17
0.02
0.11
Uni(f=2.0)
0.25
0.27
0.29
0.27
0.39
0.06
0.24
7-reweight
0.61
0.53
0.57
0.65
0.69
0.65
0.67
DiP(o=0.3)
0.50
0.41
0.45
0.53
0.55
0.54
0.52
Dist.
DiP(a=0.4)
0.59
0.52
0.57
0.66
0.70
0.64
0.66
free
ALIGNED-IS
0.90
0.75
0.80
0.89
0.91
0.85
0.87
21

---

Table 29: Robustness comparison of watermarking methods for Dolly CW with SpeechGPT under
signal processing attacks. We report TPR at 1% FPR.
No
Echo
Gauss, noise
Lowpass (40%)
Smooth (6 samp.)
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(0.9)
(1.1)
KGW()=1.0)
0.32
0.28
0.28
0.31
0.29
0.03
0.03
KGW(6=1.5)
0.56
0.49
0.48
0.57
0,58
0.06
0.04
KGW(0=2.0)
0.73
0.66
0.67
0.73
0.73
0.10
0.05
Dist.
Uni(カ=1.0)
0.17
0.19
0.16
0.17
0.18
0.12
0.09
Uni(6=1.5)
0.40
0.44
0.32
0.40
0.43
0.29
0.17
Uni(A=2.0)
0.57
0.57
0,55
0.57
0.58
0.39
0.24
y-rewcight
0.37
0.21
0.32
0.34
0.31
0.02
0.02
DiP(a=0.3)
0.23
0.15
0.19
0.23
0.21
0.02
0,02
Dist.
DiP(a=0.4)
0.29
0.19
0.30
0.28
0.29
0.01
0.00
free
ALIGNED-IS
0.82
0.78
0.81
0.81
0.80
0.23
0.15
Table 30: Robustness comparison of watermarking methods for Dolly CW with SpeechGPT under
codec-based and quantizing attacks. We report TPR at % FPR.
EnCodee
MP3
MP3
Opus (16kbps)
Opus (31kbps)
Quant.
Watermark
(24kHz))
(32kbps)
(40kbps)
(64-bit)
KGW(d=1.0)
0.26
0.16
0.16
0.29
0.30
0.14
KGW(6=1.5)
0.52
0.26
0.26
0.56
0.56
0.36
KGW(6=2.0)
0.67
0.45
0.43
0.71
0.53
Dist.
Uni(š=1.0)
0.17
0.15
0.15
0.19
0.18
0.06
Uni(5=1.5)
0.35
0.28
0.29
0.41
0.40
0.12
Uni(8=2.0)
0.55
0.49
0.46
0.61
0.60
0.23
7-reweight
0.32
0.16
0.18
0.31
0.35
0.23
DiP(a=0.3)
0.22
0.10
0.10
0.19
0,24
0.12
Dist.
DiP(a=0.4)
0.27
0.12
0.14
0,24
0.30
0.20
free
ALIGNED-IS
0.78
0.64
0.60
0.78
0.82
0.62
Table 31: Robustness comparison of watermarking methods for Finance QA with SpeechGPT under signal processing attacks. We report TPR at 1% FPR.
No
Echo
Gauss. noise
Lowpass (40%)
Smooth (6 samp.)
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(0.9)
(1.1)
KGW(#=1.0)
0.36
0.30
0.34
0.37
0.35
0.05
0.03
KGW(8=1.55
0.68
0.65
0.69
0.70
0.71
0.07
0.07
KGW(5=2.0)
0.80
0.78
0.78
0.81
0.83
0.10
0.07
Dist.
Uni(6=1.0)
0.21
0.27
0.14
0.19
0.21
0.20
0.15
Uni(
0.64
0.70
0.56
0.60
0.64
0.40
0.31
Uni(A=2.0)
0.72
0.72
0.71
0.74
0.76
0.52
0.39
7-reweight
0.56
0.37
0.51
0.52
0.55
0.03
0.04
DiP(a=0.3)
0.33
0.26
0.39
0.39
0.33
0.01
0.00
Dist.
DiP(a=0.4)
0.54
0.36
0.49
0.51
0.48
0.02
0.03
free
ALIGNED-IS
0.94
0.94
0.94
0.94
0.93
0.38
0.24
22

---

Table 32: Robustness comparison of watermarking methods for Finance QA with SpeechGPT under
codec-based and quantizing attacks. We report TPR at I % FPR.
EnCodec
MP3
MP3
Opus (16kbps)
Opus (3 (3)b)))
Quant.
Watermark
(24kHT
(32kbps)
(40kbps)
(64-bit)
KGW(d=1.0)
0.36
0.17
0.18
0.35
0.39
0.16
KGW(d=1.5)
0.69
0.38
0,40
0.71
0.69
0.50
KGW((=2.00
0.75
0.58
0.61
0.80
0.82
0.66
Dist.
Uni(Ō=1.0)
0.21
0.16
0.15
0.23
0,25
0.05
Uni(8=1.55
0.61
0.50
0.51
0.71
0.66
0.22
Uni(8=2.0)
0.71
0.63
0.65
0.77
0.75
0.44
7-reweight
0.52
0.32
0.30
0.47
0.56
0.37
DiP(a=0.3)
0.38
0.19
0.19
0.31
0.36
0.23
Dist.
DiP(a=0.4)
0.45
0.28
0,26
0.47
0.49
0.31
free
ALIGNED-IS
0.91
0.89
0.88
0.95
0.94
0.87
Table 33: Robustness comparison of watermarking methods for Longform QA with SpeechGPT
under signal processing attacks. We report TPR at 1% FPR.
No
Echo
Gauss. noise
Lowpass (40%)
Smooth
Speed
Speed
Watermark
attack
(0.05sec)
(30dB)
(6 samp.)
(0.9)
(1.1)
KGW(6=1.00 1.0)
0.40
0.38
0.38
0.44
0.38
0.06
0.01
KGW($=1.5)
0.72
0.57
0.59
0.75
0.63
0.08
0.04
KGW(6==.00)
0.84
0.83
0.81
0.83
0.83
0.10
0.08
Dist.
Uni(f=1.0)
0.25
0.30
0.22
0.23
0.29
0.15
0.10
Uni(f=1.5)
0.55
0.61
0,48
0.54
0.60
0.38
0.27
Uni(6=2.0)
0.71
0.75
0.67
0.72
0.71
0.53
0.32
--reweight
0.46
0.26
0.39
0.48
0.41
0.04
0.02
DiP(a=0.3)
0.37
0.25
0.33
0.32
0.36
0.04
0.03
Dist.
DiP(a=0.4)
0.46
0.33
0.40
0.46
0.37
0.01
0.01
free
ALIGNED-IS
0.95
0.93
0.93
0.95
0.92
0.38
0.22
Table 34: Robustness comparison of watermarking methods for Longform QA with SpeechGPT
under codec-based and quantizing attacks. We report TPR at 1% FPR.
EnCodee
MP3
MP3
Opus (16kbps)
Opus (31kbps)
Quant.
Watermark
(24kHz)
(32kbps)
(40kbps)
(64-bit)
KGW(#=1.0)
0.40
0.21
0.17
0.37
0.42
0.19
KGW($=1.5)
0.63
0.38
0.42
0.63
0.70
0.41
KGW(5=2.0)
0.80
0.59
0.61
0.84
0.84
0.64
Dist.
Uni(6=0.5)
0.07
0.05
0.03
0.08
0.08
0.04
Uni(6=1.0)
0.21
0.20
0.21
0.29
0.24
0.05
Uni(6=1.5)
0.54
0.49
0.46
0.60
0.58
0.14
Uni(8=2.0)
0.69
0.61
0.64
0.77
0.73
0.35
g-reweight
0.40
0.26
0.25
0.48
0.30
DiP(o=0.3)
0.34
0.20
0.16
0,26
0.37
0.17
Dist.
DiP(a=0.4)
0.43
0.19
0.16
0.37
0.37
0.27
free
ALIGNED-IS
0.91
0.82
0.84
0.95
0.96
0.85
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We provide the full set of assumptions and a complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper in the experimental section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We provide code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
stipaiisssooooiocyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper in the experimental section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the watermarking detectability results using hypothesis test and report the false positiva rate under rigorous statistical guarantee.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See our experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the introduction section and the broader impact section appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't release new data/models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the data/models used in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new asset is introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
30


Reviews:
[{"review_id": "zxc1lk8d1f", "reviewer": "Reviewer_g2s4", "summary": "ALIGNED-IS proposes a watermarking method for AR audio generation models. Its main contribution is achieving distortion-free watermarking despite the retokenization mismatch problem that affects direct token-based methods. To address this, the method clusters similar audio tokens and embeds watermarks at the cluster level. \n\nThe main algorithmic contribution is Aligned Inverse Sampling (AIS), a sampling strategy that aligns cluster-level probabilities with fixed intervals for watermark detection, ensuring robustness without altering the model's output distribution.\n\nAIS is proven to be statistically identical (in expectation) to the original model's distribution over clusters. Experiments show that AIS outperforms prior distortion-free methods in detectability and robustness.", "questions": "- Since AIS hinges on clustering, can you clarify why clustering is the right abstraction for addressing retokenization mismatch in AR audio models? Were other approaches considered?\n- Cluster-level watermarking likely reduces bit capacity (Def 2). Can you quantify this and discuss the impact on latency, detectability, or real-world throughput?\n- Can you evaluate AIS under adversarial conditions particularly attacks that manipulate cluster boundaries e.g., pitch shift or adaptive retokenization?\n\nNOTE: I am at the borderline and will evaluate based on the rebuttal.", "limitations": "No. The authors do not include any explicit discussion of limitations or potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper introduces a novel Aligned Inverse Sampling (AIS) thats tailored to the challenges of watermarking AR audio models, addressing the unique retokenization mismatch problem that prior methods overlook.\n- The method is theoretically grounded (Theorem 4.1) and empirically validated on audio perturbations and outperforms prior distortion-free baselines.\n- Generally, the paper is well-structured and technical components like AIS and clustering are explained with reasonably.\n- Its a timely contribution toward trustworthy content attribution in generative systems esp where provenance is important. \n\nWeaknesses\n- AIS requires access to token embeddings for clustering, and cant be used in black-box system scenarios. \n- Limited to AR models, which narrows applicability\n- Cluster-level watermarking reduces bit capacity per token, but the paper does not quantify this tradeoff.\n- The paper does not evaluate/acknowledge adversarial attacks, which is important especially for real-time audio, including adversarial stream injection attacks.", "was_revised": true, "final_justification": "Overall, I am satisfied with the rebuttal. While the use of clustering is a design choice, the empirical evidence suggests it is a valid one for this problem. However, it still carries inherent limitations, so I am keeping my recommendation at borderline accept.", "rebuttal": "Thank you for your thoughtful and positive review. We greatly appreciate your recognition of our contributions, both in addressing the retokenization mismatch challenge and in advancing trustworthy content attribution for generative audio systems. Below we address the concerns.\n\n> W1. AIS requires access to token embeddings for clustering, and cant be used in black-box system scenarios.\n\n**A1.** Clustering does not need to be performed online. The service providers run the clustering once, offline, using any model for which they have embedding access, and then publish the deterministic token→cluster lookup table (the only thing the generator/detector needs at run time). At deployment, both watermarking and detection use only token IDs and the precomputed mapping, so no embedding access is required in the black‑box setting. This is identical in spirit to prior watermarking schemes that precompute hash-based green/red lists and only apply light-weight reweighting at inference.\nEven in the extreme case, where the token embeddings are completely inaccessible, we can still identify mismatched token pairs by comparing the original token sequence with the one obtained after a decode-then-encode roundtrip. Using these mismatches, we can construct a graph where nodes represent tokens and edges connect mismatched pairs. The connected components of this graph naturally define clusters that reflect the model’s retokenization behavior, allowing our method to adapt without relying on token embeddings.\n\n> W2. Limited to AR models, which narrows applicability\n\n**A2.** Our goal and evaluation scope is explicitly AR models, which is one of the most powerful categories of speech generation systems. That said, the core idea (cluster the discrete tokens space and watermark at the cluster level) naturally extends to any model that emits a sequence of discrete tokens. For fully non-AR paradigms (e.g., diffusion or masked LM decoding), one can instantiate the same idea over iterative refinement steps (treating each step as a “pseudo time step”) or over latent-code clusters (for VQ-tokenized models). We will make this scope explicit and briefly sketch these adaptations.\n\n> W3. Cluster-level watermarking reduces bit capacity per token, but the paper does not quantify this tradeoff.\n\n**A3.** Thank you for pointing this out. We agree that cluster-level watermarking reduces the bit capacity per token compared to token-level watermarking schemes. This trade-off is an intentional design choice to mitigate retokenization mismatch, which is particularly problematic in auto-regressive audio generation models. While token-level schemes can encode up to 1 bit per token (e.g., by selecting from two disjoint sets per token), cluster-level watermarking limits the number of distinct watermark codes to the number of clusters $h$, leading to roughly $h/|V|$ bit per token (|V|) is the volume of the vocabulary, thus reducing the number of independent bits that can be embedded across a generation.\n\nTo make this concrete, we consider the following example. Suppose we generate a sequence of $L=100$ tokens:\nIn token-level watermarking, each token can carry 1 bit. If there is no token collision, this will result in a theoretical capacity of 100 bits. In cluster-level watermarking with $h=20$ clusters, each cluster can be assigned a watermark bit, and all tokens within that cluster share the same bit. If the 100 tokens span roughly 20 clusters, the maximum number of independent watermark bits is bounded by 20.\n\n| Method | Bit Capacity per Token | Max Bits in 100 Tokens | Comments |\n| --- | -- | --- | -- |\n| Token-level watermark | \\~1.0 bit | 100 bits | High capacity, but prone to retokenization mismatch |\n| Cluster-level (h=20) | ≤0.2 bits | ≤20 bits | Lower capacity, but more robust to retokenization |\n\nDespite the lower bit capacity, our empirical results show that ALIGNED-IS achieves strong detectability while maintaining generation quality, which demonstrates that the reduced capacity is sufficient in practice. We will include this discussion and quantification in the revised version.\n\n> W4. The paper does not evaluate/acknowledge adversarial attacks, which is important especially for real-time audio, including adversarial stream injection attacks.\n\n**A4.** We kindly point the reviewer to Section 5.2, where we provide a comprehensive evaluation of AIS under 13 common attacks in audio watermarks, including common signal processing modifications, popular audio codings, and the denoising attack. Following the authors suggestion, we consider one stream injection attack: The threat model is that the attacker cuts the stream at time t and inserts tokens/frames generated by an unwatermarked model, then resumes. Below, we report the detection results of AIS with different injection strength, where $\\epsilon$ is the injection ratio (i.e., injection length / original length). We see that AIS can achieve good detectability under 30% stream injection attack.\n\nSpiritLM, Librispeech\n| $\\epsilon$  | TPR@FPR=1% |\n|-:|--:|\n| 0.1  | 0.9036  |\n| 0.2 | 0.8319 |\n| 0.3  | 0.7482 |\n\nSpeechGPT, LongformQA\n| $\\epsilon$  | TPR@FPR=1% |\n|--:|---:|\n| 0.1 | 0.9247 |\n| 0.2| 0.8593 |\n| 0.3  | 0.7735 |\n\n> Q1. Since AIS hinges on clustering, can you clarify why clustering is the right abstraction for addressing retokenization mismatch in AR audio models? Were other approaches considered?\n\n**A.** To quantify the effectiveness of our clustering method in mitigating retokenization mismatch, we report the token mismatch rates before and after applying clustering on SpiritLM across multiple datasets. The mismatch rate is computed by comparing the original response tokens with those obtained after a decode-then-encode retokenization process. As shown in the table, our clustering method significantly reduces the mismatch rate, demonstrating its effectiveness in aligning tokens during watermarking generation and detection. \n\n| Dataset | Mismatch Rate Before | Mismatch Rate After | Reduction (%) |\n|---|-----|---|----|\n| mmw_book_report| 0.3749 | 0.2117 | 43.55% |\n| mmw_story | 0.3652 | 0.2174 | 40.47% |\n| mmw_fake_news | 0.4295 | 0.2300 | 46.44% |\n| dolly_cw | 0.3634 | 0.2134 | 41.30% |\n| longform_qa | 0.3757 | 0.2109 | 43.85% |\n| finance_qa | 0.3587 | 0.2133 | 40.54% |\n\nWe have also evaluated other popular clustering methods, e.g., (Gaussian mixture models, spectral clustering) and found that they produce similar results. We did not explore methods beyond clustering, as it is the most natural and direct approach to address the retokenization mismatch. Empirically, clustering has proven effective in significantly reducing such mismatches.\n\n> Q2. Cluster-level watermarking likely reduces bit capacity (Def 2). Can you quantify this and discuss the impact on latency, detectability, or real-world throughput?\n\n**A.** The quantification of bit capacity is in **A3**.\n\n**Impact on latency and throughput:** Our method introduces minimal latency overhead, as the clustering is performed once offline on the tokenizer’s vocabulary. At inference time, the only additional cost is mapping each token to its corresponding cluster, which is a simple hash lookup. Thus, real-world throughput is essentially unaffected.\n\n**Impact on detectability:** While cluster-level watermarking reduces the number of independent bits, it significantly improves alignment under decode-then-encode mismatch, which is crucial in modalities like audio or image generation. As shown in our experiments, ALIGNED-IS achieves strong detectability despite reduced capacity, indicating that the trade-off is well-balanced for practical deployment.\n\n> Q3. Can you evaluate AIS under adversarial conditions particularly attacks that manipulate cluster boundaries e.g., pitch shift or adaptive retokenization?\n\n**A.** We have evaluated AIS under a variety of adversarial conditions that can manipulate cluster boundaries. As shown in Tables 4–5 and 19–28, AIS and other baselines are tested against a range of perturbations including Speed Perturbation, EnCodec re-encoding, Gaussian Noise, and Smoothing. These attacks alter temporal or spectral properties of the waveform in ways that can shift token assignments across clusters. In particular, speed perturbation effectively causes pitch and timing shifts, while EnCodec and Gaussian noise introduce quantization and feature distortions, each capable of disrupting the token cluster boundaries. These evaluations demonstrate the robustness of AIS in realistic scenarios where adaptive retokenization effects are present. We also provided one additional pitch shift attack and two low-bitrate-based EnCodec attacks on SpeechGPT and LongformQA datasets to provide further evidence: \n\nSpeechGPT, LongformQA\n\n| Pitch Shift Attack     | TPR@FPR=1% |\n|------------------|-------------|\n| +1 Semitone      | 0.9171     |\n| -1 Semitone      | 0.9215      |\n\n| EnCoder Attack     | TPR@FPR=1% |\n|------------------|-------------|\n| FACodec [1]  (1.6 kbps) | 0.9268 |\n| Speech tokenizer [2]  (4 kbps)| 0.9085 |\n\n\n> L1. The authors do not include any explicit discussion of limitations or potential negative societal impact.\n\n**A.** We kindly point the reviewer to Appendix A, where we discuss the limitations of our work. \n\nPotential negative societal impact: While our method improves the robustness and detectability of watermarks in autoregressive audio generation, it may also raise concerns regarding surveillance and misuse. For instance, persistent watermarking of generated speech, especially when applied without user consent, could be exploited to trace or deanonymize users in voice-based applications. We will put a detailed discussion in our revision.\n\n[1] Ju, Zeqian, et al. \"Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models.\" arXiv preprint arXiv:2403.03100 (2024).\n\n[2] Zhang, Xin, et al. \"Speechtokenizer: Unified speech tokenizer for speech large language models.\" ICLR 2024."}, {"review_id": "ARXRuurbvi", "reviewer": "Reviewer_9Gaf", "summary": "This paper introduces ALIGNED-IS, a distortion-free statistical watermarking method tailored for autoregressive audio generation models. It tackles the critical retokenization mismatch issue arising from the non-deterministic nature of tokenization and re-tokenization in audio models. The core idea is to cluster similar tokens, based on continuous audio feature similarity, so that during both generation and detection, tokens within the same cluster are treated as equivalent. This cluster-level sampling is paired with a newly proposed Aligned Inverse Sampling strategy to embed watermarks without distorting model output distributions. Extensive experiments demonstrate the method’s competitive detectability, robustness under audio attacks, and minimal impact on audio quality.", "questions": "1. Could the authors provide quantitative analysis on how clustering alleviates mismatch? For example, what is the match rate before and after clustering, or the percentage of mismatched tokens still falling into the same cluster?\n2. What is the computational overhead of the clustering step (once per model) and aligned inverse sampling (per generation)? A comparison to KGW or γ-reweight would be helpful.\n3. Could the authors report detection performance across different numbers of clusters (e.g., h=10, 20, 50) and include a non-clustering baseline?\n4. Can the clustering-based approach generalize to other generation models (e.g., multimodal or image) where feature similarity might not correspond to token similarity?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The method tackles a fundamental limitation, retokenization mismatch, in applying statistical watermarks to audio generation models. The use of token clustering and aligned inverse sampling is both effective and intuitive.\n\n- The authors test on multiple audio models (SpeechGPT, SpiritLM) and datasets, comparing detectability, robustness, and fidelity against state-of-the-art methods.\n\n---\n\nWeaknesses\n\n- Writing quality: Minor writing issues are present (e.g., lines 42–45 are repeated verbatim).\n\n- Insufficient analysis on clustering effectiveness: The paper does not quantify how much the clustering mitigates retokenization mismatch. Metrics like how well tokens match before and after clustering, or how often mismatched tokens end up in the same cluster, could help better support the claims.\n- Limited practical advantage: Compared to KGW(δ=1.5), ALIGNED-IS offers similar detectability and robustness, but with extra preprocessing (clustering) and sampling overhead, which may limit practical deployment. Arguably, its only advantage is better fidelity, but this advantage is not particularly significant.\n- Generalization and scalability concerns: It remains unclear whether similar clustering properties hold for other modalities (e.g., image models) or more audio models. The paper could benefit from discussion or experiments on extensibility.", "was_revised": true, "final_justification": "The response addressed my concerns. However, the technique contribution is somewhat limited. Thus, I slightly raise my score.", "rebuttal": "Thank you for the detailed and thoughtful comments, we're glad that you found our approach to retokenization mismatch both effective and intuitive. Below we address the concerns.\n\n> W1. Writing quality: Minor writing issues are present (e.g., lines 42–45 are repeated verbatim).\n\n**A1.** Thank you for pointing out, we will fix it in our revision.\n\n> W2. Insufficient analysis on clustering effectiveness: The paper does not quantify how much the clustering mitigates retokenization mismatch. Metrics like how well tokens match before and after clustering, or how often mismatched tokens end up in the same cluster, could help better support the claims.\n\n**A2.** Thank you for the suggestion. To quantify the effectiveness of our clustering method in mitigating retokenization mismatch, we report the token mismatch rates before and after applying clustering on SpiritLM across multiple datasets. The mismatch rate is computed by comparing the original response tokens with those obtained after a decode-then-encode retokenization process. As shown in the table, our clustering method significantly reduces the mismatch rate, demonstrating its effectiveness in aligning tokens and preserving fidelity during generation.\n\n| Dataset | Mismatch Rate Before | Mismatch Rate After | Reduction (%) |\n|----------------|----------------------|---------------------|---------------------|\n| mmw_book_report| 0.3749 | 0.2117 | 43.55% |\n| mmw_story | 0.3652 | 0.2174 | 40.47% |\n| mmw_fake_news | 0.4295 | 0.2300 | 46.44% |\n| dolly_cw | 0.3634 | 0.2134 | 41.30% |\n| longform_qa | 0.3757 | 0.2109 | 43.85% |\n| finance_qa | 0.3587 | 0.2133 | 40.54% |\n\n> W3. Limited practical advantage: Compared to KGW(δ=1.5), ALIGNED-IS offers similar detectability and robustness, but with extra preprocessing (clustering) and sampling overhead, which may limit practical deployment. Arguably, its only advantage is better fidelity, but this advantage is not particularly significant.\n\n**A3.** We respectfully disagree with this claim. While ALIGNED-IS introduces a **one-time** clustering overhead, it offers a key practical advantage over KGW(δ=1.5) by better preserving the original model’s distribution. As shown in Figure 2, KGW(δ=1.5) noticeably degrades generation quality (e.g., higher FAD scores), whereas ALIGNED-IS maintains fidelity. This fidelity preservation is critical for service providers who require watermarking without compromising user experience. Furthermore, based on [1], we evaluated additional quality metrics: speaker similarity  and ASR-WER, and observed that ALIGNED-IS consistently outperforms KGW(δ=1.5) in these aspects, reinforcing its practical value beyond detectability and robustness. Baseline is the content generated without watermark.\n\n| Method      | Speaker Similarity ↑ | ASR-WER ↓ |\n|-------------|-----------------------|-----------|\n| Baseline    | 0.7550        | 0.1475    |\n| Aligned-IS  | 0.7548     | 0.1472    |\n| KGW(1.5) | 0.7446   |  0.1617    |\n\nBesides, the one-time cost of the clustering step is 0.3935s for Spiritlm and 0.4150s for Speechgpt. For audio generation, both ALIGNED-IS and KGW have a computational complexity of $O(VL)$, where $V$ denotes the size of the token set and $L$ is the generation length. Empirically, ALIGNED-IS and KGW introduce negligible (<0.01 s/token) computational overhead to the generation process. Thus, they have similar time efficiency.\n\n> W4. Generalization and scalability concerns: It remains unclear whether similar clustering properties hold for other modalities (e.g., image models) or more audio models. The paper could benefit from discussion or experiments on extensibility.\n\n**A4.** Our approach is indeed extensible beyond audio models. For example, autoregressive image generation models such as Emu3 [2] also employ an image-token encoder-decoder structure, analogous to audio tokenization. We conducted a clustering analysis on Emu3 and observed similar retokenization mismatch patterns, which our clustering method effectively mitigates (see the table below). This demonstrates that the underlying principles of our approach generalize to other modalities that rely on discrete token representations, supporting its broader applicability and scalability.\nIn this table, we calculated the retokenization mismatch rate with and without clustering method on Emu3 with 500 images from LAION dataset:\n\n| Mismatch Rate Before | Mismatch Rate After | Reduction (%) |\n|-------|------|-----|\n| 0.5131 | 0.2305 | 55.07% |\n\n\n\n> Q1. Could the authors provide quantitative analysis on how clustering alleviates mismatch? For example, what is the match rate before and after clustering, or the percentage of mismatched tokens still falling into the same cluster?\n\n**A.** See **A2**.\n\n> Q2. What is the computational overhead of the clustering step (once per model) and aligned inverse sampling (per generation)? A comparison to KGW or γ-reweight would be helpful.\n\n**A.** The time cost of the clustering step is 0.3935s for Spiritlm and 0.4150s for Speechgpt. The clustering method does not cost a lot because 1) we use the basic k-means clustering and 2) the number of speech tokens is small (500~1000). For audio generation, ALIGNED-IS, KGW, and γ-reweight all have a computational complexity of $O(VL)$, where $V$ is the size of the token set and $L$ is the generation length. Similar to KGW and γ-reweight, our method add negligible (<0.01 s/token) computational overhead to the generation process.\n\nQ3. Could the authors report detection performance across different numbers of clusters (e.g., h=10, 20, 50) and include a non-clustering baseline?\n\n**A.** We conducted an ablation study on the number of clusters for SpiritLM using the Dolly CW dataset, with generation settings identical to the main experiments. The results show that detectability initially increases with more clusters but begins to decline beyond a certain point. This trend arises because SpiritLM has approximately 500 audio tokens, and using too many clusters leads to overly fine partitions that fail to effectively mitigate retokenization errors. We find that $h=20$ strikes a good balance and yields optimal detectability, supporting our choice in the main experiments. For the non-clustering baseline, we assign tokens randomly to h=20 clusters and apply ALIGNED-IS.\n\n|  h  |   TPR@FPR=1%   | Median p-value |\n|:--:|:------:|:--------:|\n| non-cluster | 0.6112 | 0.0059 |\n| 10 | 0.7983 | 0.00047 |\n| 20 | 0.9220 | 2.27e-05 |\n| 40 | 0.9044 | 6.96e-05 |\n| 100 | 0.8586 | 0.00026 |\n| 150 | 0.6879 | 0.0029 |\n\nQ4. Can the clustering-based approach generalize to other generation models (e.g., multimodal or image) where feature similarity might not correspond to token similarity?\n\n**A.** Yes, the clustering-based approach can generalize to other generation models, including multimodal or image models, even when feature similarity does not align with token similarity. In such cases, we can identify mismatched token pairs by comparing the original token sequence with the one obtained after a decode-then-encode roundtrip. Using these mismatches, we construct a graph where nodes represent tokens and edges connect mismatched pairs. The connected components of this graph naturally define clusters that reflect the model’s retokenization behavior, allowing our method to adapt without relying on direct feature similarity.\n\n[1]  O’Reilly et al. Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech. ICLR GenAI workshop.\n\n[2] Wang et al. Emu3: Next-Token Prediction is All You Need, https://arxiv.org/abs/2409.18869."}, {"review_id": "Z9TuebyG8X", "reviewer": "Reviewer_Hewn", "summary": "The authors propose a language model-style watermarking scheme for autoregressive generative models operating on audio tokens. Existing approaches struggle due to the fact that audio tokenizers lack cycle consistency / idempotence, meaning that decoding tokens to audio and re-encoding results in substantial changes to the tokens, thereby thwarting existing statistical watermark detection schemes. The proposed method uses clustering based on pre-quantization encoded features to circumvent this issue. The authors demonstrate that via an Aligned ITS scheme, the proposed method can generate robustly detectable watermarked audio by guiding sampling at the cluster level rather than the token level.", "questions": "The paper checklist seems to indicate that code is provided in a supplement, but I can not find any supplementary material beyond the paper's appendix.\n\nThe authors claim 0% detection rates for AudioSeal under processing with the original Encodec, which is surprising given the Encodec experiments in the original AudioSeal paper and the results from O'Reilly et al. (https://arxiv.org/abs/2504.10782). The 12kHz bandwidth seems unlikely to have an effect here as AudioSeal has an 8kHz bandwidth (16kHz sample rate), so I'm wondering if there might be a bug? Or perhaps the 12kHz Encodec is just much more effective at removing watermarks than the full-bandwidth version.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths__\n\n* Adapting \"semantic\" / \"generative\" / \"intrinsic\" watermarks from the text modality to other modalities such as audio is an important research direction -- particularly in audio, where imperceptible post-hoc watermarks currently dominate but have been shown to be fragile against re-generation attacks.\n\n* The proposed method does not require training a robust tokenizer to circumvent the \"retokenization mismatch\" issue, and is therefore widely applicable to existing autoregressive audio generative models\n\n* The robustness evaluations indicate the proposed method may improve on the robustness of post-hoc audio watermarks under certain classes of transformation\n\n* The proposed Aligned-ITS scheme is interesting, and detection only relies on the assumption that the conditional probabilities of token clusters are approximately equal ($\\approx \\frac{1}{h}$)\n\n\n\n__Weaknesses__\n\n* The authors do not appear to mention the length of generated audio required to detect the watermark. The authors need to clarify the average length of generated audio used in their detection experiments, and ideally conduct an additional experiment demonstrating detection performance as a function of generation length. Readers will want to know: \"Can I detect this watermark in a 5-second random excerpt of generated audio? In a 2-second excerpt?\"\n\n* While the proposed watermark is \"distortion free\" and should in theory produce high quality audio, the audio quality evaluations are unconvincing:\n  * The use of FAD is suspect. AudioSeal performs poorly on FAD compared to the proposed method despite being essentially human-imperceptible, raising questions about the alignment of this metric with human perception. The authors do not state what underlying embedding model is used to compute FAD. If these embeddings are computed on e.g. SSL features at a fixed small hop length, then note that a \"watermark\" that chops and randomly re-shuffles audio could achieve near-0 FAD because the underlying frame-level embedding sets would be the same, despite the watermark destroying audio quality.\n  * The authors do not use standard metrics for evaluating the output of autoregressive speech language models such as ASR word/character error rates (for intelligibility) and speaker similarity (if a speaker audio prompt is provided). These are much more interpretable than automated \"MOS\" evaluations, and basically standard -- for instance, the paper for the SpiritLM model used by the authors evaluates ASR CER, the VALL-E paper evaluates ASR WER and speaker similarity, etc.\n  * The authors do not conduct a human listening evaluation or provide audio examples of their watermark. This makes it difficult to trust that the proposed method does not noticeably deteriorate generated audio quality. The authors should certainly provide listening examples\n\n* The robustness of the proposed method against desynchronization is not clear. It performs most poorly against speed change, which indicates a potential issue in which a fixed-frame-rate audio tokenizer results in \"drift\" between token streams from the original generated audio and generated audio that has been shifted in time, time-stretched, or had its speed changed. The authors could address this with some very simple experiments:\n  * Akin to Figure 7 in the WavMark paper, evaluate detection performance as a function of time shift (but include both positive and negative shifts). This should address the concern over whether small shift of e.g. 25% or 50% of the tokenizer frame length results in tokens being mapped to a different cluster (thus destroying the watermark)\n  * For different speed change amounts, plot token-level detection scores as a function of token sequence position. As we move farther along in the watermarked audio, do our detection scores fall off as the speed change causes the detector to desynchronize more and more?\n\n* The authors only evaluate robustness to one codec-based regeneration attack (the original Encodec). Works such as O'Reilly et al. (https://arxiv.org/abs/2504.10782) have shown this to be much weaker at removing watermarks than recent low-bitrate codecs. It would nice to see the authors evaluate a couple of more recent low-bitrate codec attacks, as this is where even stronger post-hoc watermarks not considered by the authors (e.g. Timbre-Watermark) fail.\n\n* The general robustness of many watermarking methods evaluated by the authors (e.g. KGW, ITS) even without the proposed clustering mechanism seems to suggest that a significant portion of the observed robustness comes from the robustness of the tokenizers used rather than the clustering mechanism.\n\n* Limited scope: despite the potential generality of the proposed method, the authors only evaluate their method within two multimodal speech LLMs. This approach should in theory be applicable to TTS models (VALL-E, Parler-TTS) music generation models (MusicGen), etc. While the authors do not consider multi-level RVQ audio tokenizers, which still dominate many audio generation tasks, the proposed method should in theory still be applicable (e.g. by watermarking only the first RVQ level). Overall, I think this weakness is minor compared to the others mentioned above, and probably not feasible to address in the rebuttal period.\n\n* Related work:\n  * The \"retokenization mismatch\" referred to by the authors, including its relevance to language model-style watermarking techniques, has been covered by Défossez et al. (https://arxiv.org/abs/2410.00037, arXiv) and by O'Reilly et al. (https://arxiv.org/abs/2410.11025, ICASSP 2025).\n\n  * The authors do not address previous attempts at discrete token-based audio watermarking (e.g. \"DiscreteWM\", AAAI 2025 https://arxiv.org/abs/2412.13917) and \"generative\" or \"intrinsic\" approaches that attempt to ground watermarks within generative models (e.g. \"GROOT\", ACM MM 2024, https://arxiv.org/abs/2407.10471). While these existing approaches have potential robustness limitations (DiscreteWM) or produce \"shallow\" signatures akin to post-hoc watermarks (GROOT), they definitely merit a mention. Among attempted \"generative\" or \"intrinsic\" audio watermarks, the following publications come to mind:\n\n    * https://arxiv.org/abs/2309.15224 (ICASSP 2024)\n    * https://arxiv.org/abs/2409.13382 (ICASSP 2025)\n    * https://arxiv.org/abs/2407.10471 (ASM MM 2024)\n    * https://arxiv.org/abs/2406.04840 (Interspeech 2024)\n    * https://arxiv.org/abs/2202.08900 (ICASSP 2022)\n\n\n__Summary and Recommendations__\n\nAs a result of the above concerns, I do not currently recommend accepting the paper. __However, I am willing to raise my score provided the authors address the most pressing points mentioned above__. In order of significance, these are:\n1. The lack of audio examples and flawed audio quality evaluation (audio examples most important, followed by standard quality metrics)\n2. The lack of information on the required detection length\n3. Potential synchronization issue\n4. Lack of more recent codec attacks in robustness evaluation\n\nAddressing points 1-2 would move my rating to accept. Additionally addressing points 3-4 would provide much stronger evidence that the proposed method is actually superior to post-hoc watermarks against the most challenging attacks, and would further increase my score. Concerns re: related work should be trivial to address, and I'm assuming the authors will add missing citations or otherwise discuss why they feel these prior works are not relevant.\n\nOverall, I think this work has the potential to make a strong case for the feasibility of language model-style \"semantic\" watermarking of autoregressive audio generative models.", "was_revised": true, "final_justification": "The authors addressed the majority of my concerns in the rebuttal, including improved audio quality metrics, robustness evaluation, and desynchronization evaluation. As a result, I have raised my score.", "rebuttal": "Thank you for your very detailed, thoughtful, and constructive feedback. We appreciate your recognition of the broader importance of adapting generative watermarking methods beyond text. Below we address the concerns.\n\n> W1. The length of generated audio required to detect the watermark. \n\n**A1.** The length of audio token sequences used for detection is typically around 500, which corresponds to approximately 10–20 seconds of audio. The duration is affected by the tokenizer and vocoder frame rates. Some variation arises because the duration of each token may differ depending on the model’s duration prediction. In the table below, we present detection results on randomly cropped segments of watermarked audio with varying durations. As shown, detection performance improves with longer audio segments, achieving reliable detectability for audio durations exceeding 5 seconds.\n\nSpeechGPT/Longform QA\n\n| Time (sec) | TPR@FPR=1% |\n|---|--|\n| ~2 |0.50|\n| ~3 |0.67|\n| ~4|0.82|\n| ~5| 0.87|\n| ~6 | 0.92|\n| ~7 | 0.93|\n| ~8| 0.94|\n| ~9| 0.95 |\n\n> W2 a) The use of FAD is suspect. \n\n**A2 a).** FAD it is not explicitly aligned with human perception, the poor results of AudioSeal most likely arise from deviations in barely audible frequencies. In our paper, we used the CNN-based VGGish model to compute FAD [1], which uses non-overlapping frames and would not be robust to the reshuffling attack that the reviewer aptly points out. We thus re-evaluated the audios using the transformer-based CLAP model as the FAD feature extractor and verified our distortion-free claim.\n\n| Method  | FAD (CLAP) |\n|--|--|\n| No watermark   | 0.0036|\n| KGW ($\\delta$=2.0) | 0.0052|\n| Uni ($\\delta$=2.0) | 0.0041 |\n| $\\gamma$-reweight  | 0.0034|\n| AudioSeal  | 0.0084 |\n| WavMark | 0.0693 |\n| AlignedIS | 0.0031 |\n\n> W2 b) Evaluate audio quality under ASR word/character error rates and speaker similarity. \n\n**A2 b).** We evaluate speaker similarity using WavLM-Base [3], and ASR-CER/WER using HuBERT-large fine-tuned model [4], with watermarked audios. From the results, we observe that distortion-free watermarking methods, e.g., Aligned-IS, DiPmark, and γ-reweight, achieve performance comparable to the unwatermarked baseline across all metrics. In contrast, biased and post-hoc watermarking methods noticeably degrade the generation quality.\n\nSpiritLM/Librispeech\n| Method      | Speaker Similarity ↑ | ASR-CER ↓ | ASR-WER ↓ |\n|---|---|--|--|\n| No watermark    | 0.7550 | 0.1039 | 0.1475|\n| Audioseal | 0.7479 | 0.1086| 0.1648 |\n| Wavmark| 0.7311 | 0.1143| 0.1819|\n| Aligned-IS  | 0.7548  | 0.1038| 0.1472|\n| DiPmark(0.4)  | 0.7543 | 0.104 | 0.1468|\n| γ-reweight| 0.7546 | 0.1043| 0.1477 |\n| KGW(1.5) | 0.7446| 0.1093|  0.1617 |\n| Unigram(1.5)  | 0.7431 | 0.1102| 0.1656|\n\n\n> W2 c) The authors do not conduct a human listening evaluation or provide audio examples of their watermark. \n\n**A2 c).** According to the NeurIPS rebuttal guideline, we are not able to add links to the audio examples, but we will definitely provide audio examples in the supplementary if the paper gets accepted. To assess the quality of Aligned-IS, we conducted a human listening study with five English-speaking participants. Each participant was presented with 50 randomly ordered pairs of watermarked and unwatermarked audio samples and asked to select the one with better quality in each pair. A binomial test under the null hypothesis that both versions are equally likely to be preferred ($p = 0.5$) yielded a $p$-value of 0.73, indicating no statistically significant difference. We conclude that Aligned-IS introduces no perceptible degradation in audio quality.\n| Speaker |1|2 |3 |4 |5 | Total |\n|-|-|-|-|--|-|-|\n| Unwatermarked Win|24 |27 |28 |23 | 25|127|\n| Aligned-IS   Win |26|23|22 |27 |25 |12 |\n\n\n\n> W3. The robustness of the proposed method against desynchronization is not clear. \n\n**A3.** We acknowledge that our current watermarking scheme, which relies on a fixed-frame-rate audio tokenizer, is not inherently robust to speed-based desynchronization attacks such as time-stretching or speed perturbation. These attacks introduce temporal \"drift\" between the original and modified token frames, potentially disrupting cluster assignments and degrading detection performance. However, this limitation can be partially addressed using post-hoc alignment techniques e.g. by applying a grid search to resynchronize the modified audio with the original timing.\n\n> W3 a) Evaluate detection performance as a function of time shift.\n\n**A3 a).** Indeed, the proposed watermark is not inherently robust to time shift, but deception of the watermark would be possible by tokenization at a few various small offsets (smaller than the frame size). We performed an experiment with the Longform QA dataset and the SpiritLM model, whose frame size is 645 samples. We uniformly sampled 8 time shifts, with shifts larger than 322 being equivalent to negative shifts of less than 50% of the frame length. In the following Table, we observe our watermark’s strength decrease and then increase, but still maintaining high detectability at small offsets.\n\n| Offset (samples) | TPR@FPR=1% |\n|---|-|\n| 0   | 0.94|\n| 80 | 0.92|\n| 160| 0.76|\n| 240 | 0.58|\n| 320  | 0.43 |\n| 400 | 0.37|\n| 480| 0.54|\n| 560| 0.88|\n| 640 | 0.94|\n\n> W3 b) Plot token-level detection scores as a function of token sequence position.\n\n**A3 b).** The reviewer’s assumption is very reasonable, but it does not apply to our watermarking scheme because its positive scores are sparsely distributed in the token sequence. The sparsity is controlled by the granularity of clusters used. We plotted the average token-level scores as a function of token position but did not observe a diminishing trend.\n\n> W4. The authors only evaluate robustness to one codec-based regeneration attack (the original Encodec). \n\n**A4.** We evaluate two recent stronger codec-based attacks proposed in O'Reilly et al.: FACodec [5] and SpeechTokenizer [6]. We use the default configurations from their official GitHub repositories, corresponding to 1.6 kbps for FACodec and 4 kbps for SpeechTokenizer. As shown in the results below, Aligned-IS remains effective under both attacks, while Audioseal and WavMark fail to maintain detection performance, verifying O’Reilly et al.’s fragility conclusion.\n\nSpiritLM / Longform QA\n| TPR@FPR=1% |FACodec | SpeechTokenizer|\n|--|--|-|\n| Aligned-IS | 0.95 | 0.92 |\n| Audioseal | 0.00 | 0.00 |\n| Wavmark | 0.00 | 0.00 |\n\nSpeechGPT / Longform QA\n| TPR@FPR=1% |FACodec | SpeechTokenizer|\n|--|--|--|\n| Aligned-IS | 0.93 | 0.91 |\n| Audioseal | 0.00 | 0.00 |\n| Wavmark | 0.00 | 0.00 |\n\n> W5.  A significant portion of the observed robustness comes from the robustness of the tokenizers...\n\n**A5.** While tokenizer stability contributes to robustness, our clustering mechanism targets a distinct challenge: **retokenization mismatch** in discrete audio generation, where decoding variability alters token sequences despite perceptual similarity. Clustering mitigates this issue by grouping similar tokens, making watermark detection more resilient. It complements, rather than replaces, tokenizer robustness by addressing a modality-specific failure mode.\n\n> W6. Limited scope\n\n**A6.** We appreciate the reviewer’s suggestion. However, applying our method to models like VALL-E, Parler-TTS, or MusicGen is nontrivial due to their cross-modal nature, where input and output modalities differ. These models lack a reverse mapping from audio to token space, which is essential for verification. Supporting them would require training an additional audio-to-token module, undermining the training-free nature of our approach.\n\nDespite that, our approach is extensible beyond audio models. For example, autoregressive image generation models such as Emu3 [2] also employ an image-token encoder-decoder structure. We conducted a clustering analysis on Emu3 and observed similar retokenization mismatch patterns, which our clustering method effectively mitigates (see the table below). \n\n| Mismatch Rate Before | Mismatch Rate After | Reduction|\n|---|-|-|\n| 0.5131 | 0.2305 | 55.07% |\n\n\n> W7. Related work:\n\nA7. Thank you for highlighting the missing related work. We will incorporate a detailed discussion of these references in the revised manuscript.\n\n> Q1. Code is missing\n\n**A.** We truly apologize for the missing code. Since we are not allowed to submit links in the rebuttal, we send the anonymous github link to AC according to the NeurIPS 2025 rebuttal guidelines.\n\n> Q2. 0% detection rates for AudioSeal under original Encodec.\n\n**A.** We sincerely thank the reviewer for identifying this issue. Indeed, we examined our implementation and identified that EnCodec wrongly assumed 24 kHz sampling rate, using a hard-coded value. After fixing our code and verifying that the rest of the attacks did not contain similar issues, we re-ran the EnCodec attack under varying bitrate settings. As expected, the performance of WavMark and Audioseal remains poor across all bandwidths. While Audioseal shows an improvement as the bandwidth increases, it still fails to achieve a strong detection rate.\n\n|  Bitrate (kbps) | 3 | 6  | 12 | 24 |\n|--|--|-|--|--|\n| Audioseal detection | 0| 0 | 0| 0.05 |\n| Audioseal avg. probability | 0.3733  | 0.4413  | 0.7252  | 0.9368  |\n| Wavmark detection   | 0 | 0 | 0 | 0 |\n| AlignedIS detection | 0.8740  | 0.8940  | 0.9040  | 0.9160  |\n\n[1] Fréchet Audio Distance: A Reference-free Metric for Evaluating Music Enhancement Algorithms. Interspeech. 2019.\n\n[2] Emu3: Next-Token Prediction is All You Need, arxiv2409.18869.\n\n[3] Wavlm: Large-scale self-supervised pre-training for full stack speech processing, 2022.\n\n[4] Hubert: Self-supervised speech representation learning by masked prediction of hidden units, 2021.\n\n[5] Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv:2403.03100.\n\n[6] Speechtokenizer: Unified speech tokenizer for speech large language models. ICLR 2024."}, {"review_id": "9zYif17tLx", "reviewer": "Reviewer_pf3V", "summary": "This paper find retokenization mismatch phenomenon when applying the previous LM watermarking algorithms to autoregressive audio generation models. To address this issue, they propose ALIGNED-IS with audio token clustering and aligned inverse sampling. This method allows add distortion-free watermark to generated audio with outstanding detectability and robustness to audio coding. However, there are some concerns about the adequacy of the experiments and the statement of limitations. Please refer to the Weakness Questions, and Limitations.", "questions": "1. It may not be appropriate to use FAD alone to demonstrate audio quality, which, as it is defined, measures only divergence between two distributions.  Meanwhile, authors stated that “audios processed by WavMark have audible high-frequency artifacts” in Appendix C.2. However, they obtain higher NISQA and DNSMOSPro, as shown in Tables 14, 15, 16, and 17. Could authors clarify the potential reasons for this discrepancy and provide more experimental results to demonstrate that “post hoc audio watermarking degrade output quality”. Moreover, I’m wondering about NISQA and DNSMOSPro scores for other baselines in Figure 2.\n2. If I understand correctly, ALIGNED-IS can only convey information like \"Yes\" or \"No\". It raises the limitation that its information capacity is inferior to post-processing methods.\n3. According to the results in Appendix C, ALIGNED-IS is not robust to speed changes. It could be helpful to discuss this for a full presentation of this work.\n4. In section 3.1, the authors stated “Each code θi is typically derived from a secret key key ∈ K and the n-gram preceding context, denoted xt−n:t−1” Could you clarify this with instance or references?", "limitations": "I suggest incorporating a discussion of the limitations of application, information capacity, and robustness into Appendix A.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The “audio token clustering” substantially mitigates the retokenization mismatch problem.\n2. The aligned inverse sampling can generate distortion-free, detectable, and robust watermarks for autoregressive audio generation models.\nWeaknesses:\n1. If I understand correctly, ALIGNED-IS is only applicable to audio generation models based on LM and discrete audio tokens. They are challenging to apply to other audio generation models, such as those based on LM and diffusion, as well as those based on diffusion only, which are also popular audio generation methods.\n2. The evaluations of audio quality seem insufficient. The detailed comment is given in the Questions.\n3. The discussion of limitations is not comprehensive enough.", "was_revised": true, "final_justification": "The rebuttal provided by the authors has addressed my concerns well, and I have decided to maintain my positive evaluation.", "rebuttal": "Thank you for your positive and thoughtful feedback. We're glad you found the audio token clustering effective in addressing retokenization mismatch, and that the aligned inverse sampling approach successfully balances distortion-free generation, detectability, and robustness.\n\n> W1. If I understand correctly, ALIGNED-IS is only applicable to audio generation models based on LM and discrete audio tokens. They are challenging to apply to other audio generation models, such as those based on LM and diffusion, as well as those based on diffusion only, which are also popular audio generation methods.\n\n**A1.** Yes. ALIGNED‑IS is intentionally scoped to LM‑based, discrete‑token audio generators, a family that is widely deployed for high‑quality, low‑latency speech synthesis. The only structural requirement our method uses is a sequential stream of discrete symbols. Under that condition, the same cluster‑level watermarking mechanism carries over to any model that generates discrete tokens incrementally, including LM–diffusion hybrids that quantize audio with VQ‑codebooks (e.g., EnCodec/SoundStream‑style front‑ends) and transducer/RNN‑T style decoders. For pure diffusion pipelines that never expose discrete tokens, two straightforward adaptations exist: a) Discretize an intermediate latent via a learned codebook and apply our cluster‑level watermark there; or b) Treat each diffusion step as a pseudo time index and watermark step‑wise cluster assignments in the latent space.\n\n> W2. The discussion of limitations is not comprehensive enough.\n\n**A2.** We will refine the limitation discussion based on the reviewers’ comments and our rebuttal.\n\n> Q1. It may not be appropriate to use FAD alone to demonstrate audio quality, which, as it is defined, measures only divergence between two distributions. Meanwhile, authors stated that “audios processed by WavMark have audible high-frequency artifacts” in Appendix C.2. However, they obtain higher NISQA and DNSMOSPro, as shown in Tables 14, 15, 16, and 17. Could authors clarify the potential reasons for this discrepancy and provide more experimental results to demonstrate that “post hoc audio watermarking degrades output quality”. Moreover, I’m wondering about NISQA and DNSMOSPro scores for other baselines in Figure 2.\n\n**A.** Thank you for raising this point. While WavMark achieves higher scores on certain automated MOS metrics like NISQA and DNSMOSPro, this is likely because it is explicitly optimized for those metrics, which do not always align with perceptual audio quality, especially when artifacts such as high-frequency noise are introduced, as noted in Appendix C.2. Since automated \"MOS\" scores can be difficult to interpret meaningfully, we follow [1] and Reviewer Hewn's suggestion to include additional quality metrics such as speaker similarity, ASR-CER (character error rate), and ASR-WER (word error rate) on SpiritLM with Librispeech dataset. Across these dimensions, ALIGNED-IS consistently outperforms both biased watermarking methods (e.g., KGW, Unigram) and post-hoc watermarking approaches like WavMark, providing more robust and interpretable evidence of its superior audio fidelity in practice.\n\nSpeaker Similarity: WavLM-Base [6].  \n\nASR-CER/WER: Hubert-large, fine-tuned [7]. \n,\n| Method      | Speaker Similarity ↑ | ASR-CER ↓ | ASR-WER ↓ |\n|-------------|-----------------------|-----------|-----------|\n| Baseline    | 0.7550                | 0.1039    | 0.1475    |\n| Audioseal   | 0.7479                | 0.1086    | 0.1648    |\n| Wavmark     | 0.7311                | 0.1143    | 0.1819    |\n| Aligned-IS  | 0.7548                | 0.1038    | 0.1472    |\n| DiPmark(0.4)  | 0.7543                | 0.1045    | 0.1468   |\n| \\gamma-reweight| 0.7546                | 0.1043    | 0.1477    |\n| KGW(1.5) | 0.7446                | 0.1093    |  0.1617    |\n| Unigram(1.5)  | 0.7431                | 0.1102    | 0.1656    |\n\n> Q2. If I understand correctly, ALIGNED-IS can only convey information like \"Yes\" or \"No\". It raises the limitation that its information capacity is inferior to post-processing methods. \n\n**A.** Yes, the current algorithm focuses on zero-bit watermarks. To improve the information capacity (e.g. n-bit watermarking), one can a) get the bit expression e.g. (1011…1), b) create n+1 ALIGNED-IS which will be sequentially executed on the output token logits, c) if the i-th bit is 1, we apply the i+1-th ALIGNED-IS, if not, we skip this step. During detection, we can recover the i-th bit by the detecting results of the i+1-th ALIGNED-IS (1 for watermarked and 0 otherwise). \n\n> Q3. According to the results in Appendix C, ALIGNED-IS is not robust to speed changes. It could be helpful to discuss this for a full presentation of this work.\n\n**A.** We agree that ALIGNED-IS, like other methods relying on fixed-frame-rate audio tokenizers, is currently not robust to speed changes. Speed perturbations introduce temporal drift between the original and modified token sequences, which can disrupt cluster assignments and degrade detection. This is a known limitation of token-based watermarking schemes under desynchronization attacks. However, we believe this issue is addressable: for example, by applying a grid search to estimate and correct for the speed change, we can potentially realign the modified audio back to its original tokenization frame rate. We will include a discussion of this limitation and possible mitigation strategies in the revised manuscript to provide a more complete picture of the method's strengths and limitations.\n\n> Q4. In section 3.1, the authors stated “Each code θi is typically derived from a secret key key ∈ K and the n-gram preceding context, denoted xt−n:t−1” Could you clarify this with instance or references?\n\n**A.** The sentence refers to the standard key derivation process used in many prior watermarking schemes, e.g., [2–5], where the watermark code (or seed) is computed from a fixed secret key and the preceding n-gram context. For example, if we set $n=2$ (2-gram) and the preceding tokens are $\\texttt{[Hu221]}, \\texttt{[Hu105]}, \\texttt{[Hu142]}$, and we aim to generate the next token after $\\texttt{[Hu142]}$, the corresponding watermark code would be $(k, \\texttt{[Hu105]}~\\texttt{[Hu142]})$, where $k$ is a fixed secret key such as $k=(111000\\ldots1)$. This design allows the watermarking scheme to inject token-level perturbations in a context-dependent but reproducible way, as adopted in [2–5]. We will clarify this with an example and references in the revised manuscript.\n\n> L: I suggest incorporating a discussion of the limitations of application, information capacity, and robustness into Appendix A.\n\n**A.** We will add such discussion following the reviewer’s suggestion and the rebuttal.\n\n[1]  O’Reilly et al. Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech. ICLR GenAI workshop.\n\n[2] A watermark for large language models, ICML 2023\n\n[3] Unbiased watermark for large language model, ICLR 2024\n\n[4] DiPmark: A Stealthy, Efficient and Resilient Watermark for Large Language Models, ICML 2024.\n\n[5] Dathathri et al. Scalable watermarking for identifying large language model 395 outputs. Nature 2024.\n\n[6] Chen et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing, 2022.\n\n[7] Hsu et al. Hubert: Self-supervised speech representation learning by masked prediction of hidden units, 2021."}]

Author rebuttal:

Meta-review: The paper proposes ALIGNED-IS, a distortion-free watermarking method for autoregressive audio generation.

Strengths:
- Innovative solution to retokenization mismatch using token clustering and aligned inverse sampling.
- Demonstrates strong robustness against adversarial and codec-based attacks.
- Maintains high audio fidelity and speaker consistency across evaluations.

Weaknesses:
- Limited to AR models

I recommend acceptance, as the paper offers a novel and robust watermarking method with strong empirical results and high audio fidelity.
Decision: Accept (poster)