GROOT: Generating Robust Watermark for Diffusion-Model-Based Audio Synthesis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-28 08:29:08

Abstract: Amid the burgeoning development of generative models like diffusion models, the task of differentiating synthesized audio from its natural counterpart grows more daunting. Deepfake detection offers a viable solution to combat this challenge. Yet, this defensive measure unintentionally fuels the continued refinement of generative models. Watermarking emerges as a proactive and sustainable tactic, preemptively regulating the creation and dissemination of synthesized content. Thus, this paper, as a pioneer, proposes the generative robust audio watermarking method (Groot), presenting a paradigm for proactively supervising the synthesized audio and its source diffusion models. In this paradigm, the processes of watermark generation and audio synthesis occur simultaneously, facilitated by parameter-fixed diffusion models equipped with a dedicated encoder. The watermark embedded within the audio can subsequently be retrieved by a lightweight decoder. The experimental results highlight Groot's outstanding performance, particularly in terms of robustness, surpassing that of the leading state-of-the-art methods. Beyond its impressive resilience against individual post-processing attacks, Groot exhibits exceptional robustness when facing compound attacks, maintaining an average watermark extraction accuracy of around 95%.

Main paper: 
 introduction: 
 The advancements in generative adversarial networks (GANs) [9, 14, 15, 47] and diffusion models (DMs) [10, [35] [36] [37] have revolutionized the way of multimedia content generation. These models have significantly reduced the gap between generated content and authentic content, blurring the lines between what is real and what is artificial. To cope with the confusion on distinguishing between natural and synthesized content, deepfake detection has explored a variety of innovative approaches to widen the distinction. Regrettably, this effort not only enhances deepfake detection techniques but also drives the evolution of generative models. Generative models (GMs) learn from these distinctions and improve further.For several decades, watermarking has served as a proactive solution for protecting the intellectual property rights of multimedia content and tracing its origins. This technique continues to be relevant and effective in modern contexts, particularly for proactively combating deepfake threats and sourcing corresponding GMs. Recent research has identified post-hoc watermarking and generative watermarking as two primary strategies for tagging AI-generated content. Post-hoc watermarking involves embedding the watermark after the content's generation, making it an asynchronous Figure 1 : The diagram illustrates the process of supervising generated content through generative watermarking. The synthetic content generated via GMs by Alice will be subject to regulation, while Bob's may pose a high risk to society. process. In contrast, generative watermarking integrates the watermarking process with content synthesis, utilizing the same GM for both tasks. Specifically, in terms of generative image watermarking, one of the solutions incorporates watermarks into the weights or structure of GMs [22, 43, 46] . Another explores the transferability of watermarks [8, 23, 28, 40, 42, 45] , where the watermark is applied to the training data and consequently embedded in all generated images, leveraging its transferable characteristics.Generative image watermarking [8, 42, 43] has rapidly gained prominence among researchers due to its superior fidelity and robustness. While this field flourishes within the realm of imagery, generative audio watermarking remains underexplored and lacks similar advancements. Chen et al. [3] and Ding et al. [7] pioneered the exploration of generative steganography in speech using autoregressive models. Building on this foundation, Cho et al. [5] and Li et al. [21] introduced generative audio steganography algorithms based on GANs. Designed to address particular applications such as model attribution and coverless steganography, these algorithms operate under the assumption that the distribution of generated content occurs through lossless channels-a condition that deviates significantly from real-world scenarios. The inherent limitation in robustness constrains the ability of these algorithms to proactively regulate the utilization of generated content. Furthermore, advancements of DMs have accelerated the audio synthesis, leading to their widespread application. However, the aforementioned research has not yet explored the potential of generative watermarking based on DMs.To tackle these challenges, we proposed a generative robust audio watermarking (Groot) method tailored for diffusion-modelbased audio synthesis. By directly generating watermarked audio through DMs, we can regulate and trace the use conditions of the generated audio and its originating DMs. As illustrated in Fig. 1 , Alice's route demonstrates the conciseness but effectiveness of our proposed Groot method in comparison to the ordinary audio generation process, depicted as Bob's route. Specifically, the encoder converts the watermark into a format recognizable by DMs. Following joint optimization with a carefully designed loss function, the DMs can directly produce watermarked audio from the input watermark. A precise decoder is then employed to accurately extract the watermark from the generated audio. Our approach marries generative watermarking with proactive supervision, with the training overhead being exclusive to the encoder and decoder. This eliminates the necessity for complex retraining of DMs. Such a feature makes our method versatile and readily implementable as a plug-and-play solution for any diffusion model.In a nutshell, the contributions can be summarized as:• New Paradigm. We pioneered an exploratory investigation and proposed a generative audio watermarking technique for proactively supervising generated content and tracing its originating models. We utilized a meticulously designed watermark encoder and decoder to directly synthesize watermarked audio through the diffusion models.  
related work 2.1 text-to-speech diffusion models: 
 Text-to-speech (TTS) is a technique that synthesizes the waveform from the transcriptions. Nowadays, TTS embraces its significant performance boost by relying on the advantage of diffusion models. A complete TTS synthesis process consists of two main stages designed as deep neural networks: text-to-spectrogram [2, 12, 27, 31] and spectrogram-to-waveform [4, 11, [18] [19] [20] 26] . The methods employed for text-to-spectrogram leverage diffusion models to generate the mel-spectrogram from Gaussian noise, with text input serving as the prompt. For spectrogram-to-waveform, these approaches synthesize the waveform by utilizing the melspectrogram as conditional input to the DMs. The proposed Groot primarily leverages the spectrogram-to-waveform process for watermarking. DMs generate watermarked audio directly by taking the watermark transformed into the latent variable as input.Deep-Learning-based Watermarking With the gradual advancement of deep learning in the field of audio watermarking, an increasing number of deep-learning-based audio watermarking techniques have emerged [1, 24, 25, 30, 33] . Concretely, Chen et al. [1] utilized invertible neural networks (INNs) for embedding the watermark into audio to boost robustness. Liu et al. [24] pioneered a watermarking framework to withstand audio re-recording. To detect speech generated by AI, Roman et al. [33] devised a localized watermarking technique. In addition, Liu et al. [25] proposed timbre watermarking, aiming to defeat the voice cloning attacks.Generative Watermarking Diverging from post-hoc watermarking techniques, generative audio watermarking (or steganography) [3, 5, 7, 21] roots the watermark (or secret message) into GMs, facilitating the direct synthesis of watermarked (or stego) audio. Both [3] and [7] leverage autoregressive models to generate realistic cover speech samples. Specifically, [3] utilizes adaptive arithmetic decoding (AAD), whereas [7] employs the distribution copy method for embedding the secret message. Utilizing GANs, [21] directly generated stego audio from secret audio. For model attribution, [5] also employs GANs, which are trained to synthesize watermarked speech by incorporating a specific key and constraints. While these methods are constrained by transmission channels and limitations in model and practical application, the proposed Groot stands out as a plug-and-play generative watermarking method, enabling supervision through extracted watermarks. Moreover, it is capable of adapting to more robust scenarios. 
preliminaries: 
 The proposed Groot leverages vocoders based on DDPM [10] to generate the watermarked audio. The blurb of diffusion-modelbased vocoders about generation is described as follows.In the forward diffusion process, the normally distributed input s 𝑇 is produced by gradually adding Gaussian noise to the original audio s 0 ∼ 𝑞 𝑑𝑎𝑡𝑎 (s 0 ). It follows a discrete Markov chain {s 𝑡 } 𝑇 𝑡 =0 , which is also Gaussian distributed, that isEQUATIONwhere 𝛽 𝑡 ∈ (0, 1) is the variance scheduled at time step 𝑡, and I is an identity matrix. Let 𝛼 𝑡 = 1 -𝛽 𝑡 , 𝛼 𝑡 = 𝑡 𝑖=1 𝛼 𝑖 and 𝜖 ∼ N (0, I). For any time step of 𝑡, by re-parameterization, one has s 𝑡 = √ 𝛼 𝑡 s 0 + √ 1 -𝛼 𝑡 𝜖. The reverse denoising process generates an estimate of the audio waveform from the input s 𝑇 through a UNet-like network 𝜖 𝜃 . Given that the denoising process 𝑞(s 𝑡 -1 |s 𝑡 ) follows a Gaussian distribution, according to Bayes' theorem, we can derive 𝑞(s 𝑡 -1 |s 𝑡 ) as follows:EQUATIONUnfold this equation with Eq. 1 and combine like terms, we getEQUATIONEQUATIONThe network 𝜖 𝜃 is trained to estimate the noise 𝜖. Once it has been trained well, the estimated audio can be obtained by 𝜖 𝜃 with reparameterization:EQUATIONwhere𝛿 2 𝑡 = 1-𝛼 𝑡 -11-𝛼 𝑡 𝛽 𝑡 I, z ∼ N (0, I), and 𝑐 is the mel-spectrogram. 
methodology: 
 Our proposed method, Groot, aims to seamlessly connect the input latent variables of DMs with the generation of watermarked audio content. To achieve this, we are confronted with three primary challenges: first, designing an architecture to incorporate the watermark into the input of DMs; second, developing a training approach that encourages DMs to naturally generate watermarked content; and third, devising a reliable architecture for extracting the watermark from the generated audio.To solve the question above, solutions encompassing three main phases-watermarking, generating, and extracting-are illustrated in Fig. 2 . Firstly, an encoder E(•) is contrived to transform the watermark w into a latent variable. This transformed watermark is then combined with the origin latent variable s 𝑇 , serving as the input of DMs. The diffusion model employed is a pre-trained architecture, fixed and solely dedicated to generating audio content embedded with the watermark. Subsequently, an advanced decoder D(•) is developed to accurately extract the watermark from the synthetic audio. To ensure DMs incorporate the watermark into the audio content, we define a loss function that facilitates a joint training process. Consequently, the encoder E(•), a diffusion model(with its parameters fixed), and the decoder D(•) are simultaneously trained, guided by the criteria set forth by the loss function.Building upon the three main phases previously outlined, the subsequent section will detail the specific architectures of both the watermark encoder and decoder. Additionally, the methodologies for embedding and extracting watermarks, the approach to joint training, and the theory for watermark verification will be meticulously described, each in their respective segments.The watermark encoder aims to convert the watermark w to a latent variable 𝜎, which satisfies the distribution of DMs' input.The purpose of the watermark decoder is to distinguish features between the audio and the watermark, extracting the watermark ŵ from the watermarked audio x 0 .The watermark encoder is mainly composed of linear layers and Rectified Linear Unit (ReLU) activation functions. As depicted in Fig. 3 , outputs of Fully Connected (FC) layers need to thoroughly get through the ReLU activation function. The input size of the first FC layer, namely the length of w is configurable, while the output of the last FC layer needs to adapt for the input size of DMs. Under the specific layout of the encoder, the watermark is converted to a latent variable 𝜎 with Gaussian distribution, which can be recognized as an input for diffusion models.The watermark decoder is designed with a combination of a convolutional block (ConvBlock) and a Dense Block as illustrated in Fig. 3 . The ConvBlock consists of seven modified gated convolutional neural networks (MGCNN), each intricately designed to enhance the model's feature extraction capabilities. The architecture then transitions into a Dense Block, which is meticulously structured beginning with a FC layer, followed by a ReLU activation for nonlinear transformation. This sequence is succeeded by another FC layer, setting the stage for the final component of the model. The culmination of the process is marked by the application of a Sigmoid activation function, strategically chosen to predict the output with precision. This comprehensive configuration ensures a robust and efficient process, optimized for high-performance on robustness.The designed MGCNN represents an advanced iteration of the gated convolution neural networks (GCNN) [6] . This model is structured into two distinct branches: The first process data through In parallel, the second branch initially passes through a convolutional layer to extract features, which is immediately followed by batch normalization to ensure data standardization and improve network stability. The sequence culminates with the application of a sigmoid activation function, a strategic choice that enables the implementation of an effective gating mechanism. The gating mechanism stands as a cornerstone within the model, functioning to meticulously regulate the flow of vital information to subsequent layers. This strategic regulation is pivotal in effectively addressing and mitigating the vanishing gradient problem.Watermark embedding: During the embedding process, the Gaussian latent variable s 𝑇 , sampled from the standard Gaussian distribution, continues to serve as the input for DMs. To root the watermark into DMs, the watermark encoder E(•) is utilized to transform the watermark w into the latent variable 𝜎 before generation process:EQUATIONwhere 𝑣 = 𝑏 × 𝑙 𝑠 , 𝑏 is the batch size and 𝑙 𝑠 represents the length of the Gaussian latent variable s 𝑇 . Subsequently, the latent variable 𝜎 is superposed to s 𝑇 to acquire the final latent variable x 𝑇 for generating watermarked audio:EQUATIONThe generation process for watermarked audio x 0 is completed through the denoising process, directly derived from the final latent variable x 𝑇 . The denoising distribution of this generation process adheres to the same format of Eq. ( 3), with the input being replaced by x 𝑇 . In a consequence, the estimated audio can be obtained byEQUATIONwhere the pretrained noise prediction network 𝜖 𝜃 (x 𝑡 , 𝑡, 𝑐) is utilized to approximate the denoising distribution. Here, 𝑐 represents the mel-spectrogram, acting as a conditional input to assist in generating watermarked audio. And 𝛿 𝑡 z represents the random noise, introduced to enhance randomness in the generation process and augment the diversity of the audio. Finally, a traditional sampler is employed to generate watermarked audio step by step. The intact watermarking and generating stage are formalized in Algorithm 1.The watermark extraction process is an independent process that does not necessitate the use of diffusion and denoising process. The watermark decoder D(•) is designed to disentangle features between audios and watermarks for recovering the watermark ŵ:EQUATIONwhere 𝑢 = 𝑏 × 𝑙, 𝑏 denotes the batch size, 𝑙 denotes the length of the watermark. Our approach enables precise supervision of generated content and corresponding DMs by ensuring that the extracted watermark ŵ aligns with the embedded watermark. Essentially, through this innovative watermarking and extracting progress, the watermark becomes seamlessly integrated into the audio, maintaining its perceptual quality while ensuring content authenticity and traceability.During the inference process, the process of embedding watermarks is streamlined compared to the training process. Unlike the training procedure, where the latent variable 𝜎, derived from the watermark encoder, is added to the Gaussian latent variable s 𝑇 , for inference, 𝜎 is directly fed into the diffusion model. Thus, as a coverless watermarking technique, our method allows for direct use of the watermark w as input to the pretrained DMs, as illustrated in Fig. 2 . This can be formulated as:EQUATIONwhere G(•) symbolizes the generative progress employed by DMs.Ultimately, the pretrained watermark decoder skillfully extracts the watermark ŵ from the watermarked audio.Algorithm 1: Watermarking and Generating Stage.Input: Watermark w, Mel-spectrogram 𝑐. Output: Watermarked audio x 0 1 s 𝑇 ← s 𝑇 ∼ N (0, I); 2 𝜎 ← E(w); ⊲ w = {(w 𝑖 ), w 𝑖 ∈ {0, 1}} 𝑙 𝑖=1 3 x 𝑇 ← s 𝑇 + 𝜎; ⊲ Watermarking 4 for 𝑡 ← 𝑇 𝑖𝑛𝑓 𝑒𝑟 , ..., 1 do 5 if 𝑡 < 2 then z ← 0 else z ∼ N (0, I); 6 x 𝑡 -1 ← 1 √ 𝛼 𝑡 x 𝑡 -1-𝛼 𝑡 √ 1-𝛼 𝑡 𝜖 𝜃 (x 𝑡 , 𝑡, 𝑐) + 𝛿 𝑡 z;7 end 8 return x 0Our Groot method places a paramount emphasis on the dual objectives of maintaining high-quality watermarked audio and achieving precise extraction of watermarks. To adeptly balance these critical aspects, we have implemented a joint optimization strategy. This approach focuses on conducting gradient updates for both the watermark encoder and decoder, while strategically keeping the parameters of DMs unchanged. Regarding the quality of watermarked audio, we employ the logarithm Short-Time Fourier Transform (STFT) magnitude loss L 𝑚𝑎𝑔 (s 0 , x 0 ) as in [41] . It utilizes an 𝐿 1 norm to constrain the log-magnitude, as defined by:EQUATIONwhere || • || 1 denotes the 𝐿 1 norm, STFT(•) represents the STFT magnitudes, s 0 and x 0 represent the original audio and watermarked audio respectively. Furthermore, drawing upon [17] , we integrate the mel-spectrogram loss L 𝑚𝑒𝑙 (s 0 , x 0 ), which measures the distance between the original and watermarked audio using the 𝐿 1 norm. It is expressed as:EQUATIONwhere 𝜙 (•) denotes the function of mel-spectrogram transformation. The total loss that constrains watermarked audio quality can be computed as:EQUATIONwhere 𝜆 𝑚𝑎𝑔 and 𝜆 𝑚𝑒𝑙 are hyper-parameters for the log STFT magnitude loss and mel-spectrogram loss, respectively. Their target is to strike a balance between these loss terms.To guarantee successful watermark recovery, binary cross-entropy stand as the indispensable choice:EQUATION)Our final training loss is structured as follows, with both the watermark encoder and decoder optimized synchronously,EQUATION)where 𝜏 is the hyper-parameters for the total audio quality loss L 𝐴𝑢𝑑𝑖𝑜 , used to control the trade-off between audio quality and watermark extraction accuracy.Regulating generated contents and tracing associated DMs are achieved by verifying the existence of the watermark within the generated audio through test hypothesis [23, 43] . By assuming the watermark bit errors are independent to each other, with the previously defined watermark bits length 𝑙, the number of matching watermark bits 𝜅 follow the binomial distribution:EQUATION)where 𝜉 is the probability that needs to be tested under hypotheses.In the common use of the binomial test, the null hypothesis 𝐻 0 states the variable 𝑋 is observed with a random guess of probability 𝜉 = 0.5, whereby the model is not watermarked. The alternative hypothesis 𝐻 1 states the watermark is produced by the owner. We determine a threshold 𝑇 𝑆 with a given false positive rate (FPR) which is calculated from the probability density function under 𝐻 0 . Alternatively, if 𝜉 is sufficiently large under 𝐻 1 , the false negative rate (FNR) will be low, and FNR can be used to assess whether the chosen threshold provides a satisfactory trade-off in distinguishing between false positive cases and missed detection cases.To exemplify the hypothesis test, a successful verification in our experiments is given. In our case, the total number of samples is 768 and the length of the watermark is 𝑙=100. Then, we set an accepted FPR≤0.0037. From the experimental result, we have 𝜉=0.5442 and 𝜉=0.9969 under 𝐻 0 and 𝐻 1 , and the corresponding distributions are plotted in Fig. 4 . From the false cases in the figure, we find the threshold 𝑇 𝑆 ∈ [0.61, 0.99]. With the threshold, FNR≤0.012. Such a small FNR indicates that the detected cases with 𝜉=0.9963 are worth trusted. In simpler terms, when the watermark is 100 bits long, the extraction accuracy of 99.63% can be utilized to confirm the existence of the watermark. 
experimental results and analysis: 
 In this section, we conduct comprehensive experiments to rigorously evaluate our Groot method across several dimensions: fidelity and capacity, robustness. Moreover, we undertake a comparative analysis of Groot against the current state-of-the-art (SOTA) methods. The details of these experiments and their respective analyses are detailed below.Our experiments were carried out on LJSpeech [13] , LibriTTS [44] , and LibriSpeech [29] datasets. LJSpeech is a single-speaker English speech dataset featuring approximately 24 hours of audio data with a sample rate of 22.05 kHz. LibriTTS and LibriSpeech both are multi-speaker English datasets comprising around 584 hours of audio data recorded at a sample rate of 24kHz and approximately 1000 hours of audio recordings with a sample rate of 16kHz, respectively. In addition, we conducted a comprehensive comparative evaluation of proposed Groot against existing SOTA methods, including WavMark [1] , DeAR [24] , Au-dioSeal [33] , and TimbreWM [25] .We evaluated the performance of our method with different objective evaluation metrics. Short-Time Objective Intelligibility (STOI) [38] predicts the intelligibility of audio. Mean Opinion Score of Listening Quality Objective (MOSL) assesses audio quality based on the Perceptual Evaluation of Speech Quality [32] . We also conducted evaluation metrics using Structural Similarity Index Measure (SSIM) [39] . Moreover, Bit-wise Accuracy (ACC) is employed to evaluate watermark extraction accuracy.We conducted validation on Groot utilizing DiffWave [18] , WaveGrad [4] , and Pri-orGrad [20] . DiffWave [18] and WaveGrad [4] both are vocoders that utilize accelerated sampling and employ a gradient-based sampler akin to Langevin dynamics for audio synthesis, respectively. To boost the computational efficiency of vocoders, PriorGrad [20] leverages an adaptive prior derived from data statistics, which are conditioned on the provided conditional information. 2) Training settings. During the training process, the Adam optimizer [16] was utilized to update the parameters, and the learning rate was set to 2e-4. Regarding the hyper-parameters of audio quality loss in Eq. ( 12), we empirical set 𝜆 𝑚𝑎𝑔 = 0.7 and 𝜆 𝑚𝑒𝑙 = 0.3. All experiments were performed on the platform with Intel(R) Xeon Gold 5218R CPU and NVIDIA GeForce RTX 3090 GPU. To assess the efficacy of MGCNN as the foundational architecture of the watermark decoder, as detailed in Section 4.1, we conducted an ablation study with LJSpeech dataset by substituting all MGCNNs with conventional CNNs as a baseline. This investigation explored a range of capacities, specifically 100, 500, 1000, 2000 bits per second (bps). Fig. 6 showcases the watermark recovery accuracy employing different decoder structures, presented through a nested bar chart. The results indicate that MGCNNs outperform traditional CNNs significantly, showcasing a substantial enhancement in recovery accuracy as depicted in the bar chart. Across various capacities, the average accuracy attained with MGCNNs is an impressive 99.47%. Conversely, employing standard CNNs in the watermark decoder leads to a marked reduction to an average of just 89.31%. Specifically, at the 100 bps capacity, the accuracy of CNNs' is 11.14% lower compared to the performance with MGCNNs. DeAR [24] 0.2563 0.3904 0.2780 1.0000 TimbreWM [25] 0.0147 0.6086 0.0612 0.9998 Groot(DiffWave) 0.0005 0.1249 -0.0635 0.9969 DeAR [24] 0.1497 1.4560 0.1999 0.5007 TimbreWM [25] 0.2415 3.1842 0.3254 0.4995 2500 bps Groot(DiffWave) 0.0043 0.0861 -0.0643 0.9904 Groot(WaveGrad) 0.0438 0.0785 -0.0765 0.9970 Groot(PriorGrad) 0.0142 1.3182 0.0556 0.9981The Performance of Proposed Groot on Fidelity and Capacity. Fidelity gauges the extent to which watermarking minimally impacts the perceptibility of the original generation. We validated the fidelity of watermarked audio with the evaluation metrics exhibited in Section 5.1.2. Table 1 elaborate the experimental results on fidelity across different datasets employing DiffWave. In this context, Benchmark refers to the generated audio, whose results are compared to the Ground Truth. In scenarios involving watermarked audio, the results are presented in comparison to the Benchmark.The experimental results demonstrate that the quality of watermarked audio remains impressively high across various datasets For LJSpeech, the STOI metric consistently holds at 0.96, and the lowest MOSL surpasses 3.3326 with only minimal reduction. While a slight numerical decline is observed in the multi-speaker LibriTTS and LibriSpeech datasets, attributed to resampling, this decrease does not materially affect the overall quality of the watermarked audio. Importantly, there is no discernible downward trend in audio quality as the capacity increases, even at a capacity of 2000 bps, the evaluation metrics show only a marginal decrease. We also expanded the functionality of Groot to encompass Aishell-3 Chinese datasets [34] , undertaking experiments to assess its crosslingual fidelity and capacity. Detailed specifics are provided in the Appendix.The fidelity experiments conducted with different DMs on LJSpeech datasets are presented in Table 2 . These results reveal only a minor degradation in audio quality when employing WaveGrad across four different capacities, with the average STOI and SSIM scores remaining approximately 0.89 and 0.73, respectively. Moreover, recovery accuracy remains consistently high at 99%. In the case of PriorGrad, the average STOI and SSIM scores are 0.95 and 0.84, respectively. It sustains an exceptional accuracy rate across all capacities, averaging at 99.80%.Capacity refers to the number of watermark bits that can be embedded. To evaluate the scalability of Groot in terms of watermark capacity, experiments were performed over a range of bps: 2500, 3000, 3500, 4000, 4500, and 5000 bps. Fig. 5 visualizes the impact on extracting accuracy as the watermark capacity increases. As observed, while LibriTTS exhibits minimal accuracy decline even at 5000 bps, the other two datasets experience noticeable decreases, albeit within an acceptable margin. Notably, from 2500 to 4500 bps, the proposed approach achieves an accuracy of approximately 99% with negligible impact on audio quality.The Comparison With SOTA Methods on Fidelity and Capacity. Our evaluation commenced with a fidelity comparison between the proposed Groot and existing SOTA methods. The results are detailed in Table 3 , with the capacity settings (16, 32 , and 100 bps) conforming to the capacities reported in these SOTA methods. To effectively highlight the comparative outcomes, we introduce the prefix "D-" to denote the difference value between the watermarked audio and the original audio without watermark.The analysis of the experimental results indicates that despite AudioSeal showing marginally smaller discrepancies in evaluation metrics, Groot consistently matches its audio quality. While Wav-Mark and Groot are nearly indistinguishable in terms of STOI, Groot presents smaller variations across other metrics. In comparison with DeAR and TimbreWM, Groot showcases superior audio quality. To further affirm the superior capacity performance of Groot, we compared with DeAR and TimbreWM at 2500 bps. The results distinctly show that the watermark extraction accuracy for both DeAR and TimbreWm drastically falls to 50%, highlighting their limitation in adapting to high-capacity conditions. In stark contrast, Groot, achieves high recovery accuracies of 99.04%, 99.70%, and 99.81% across different diffusion models, all the while preserving considerably good audio quality.SOTA Methods Under Individual Attacks. To ensure the integrity and resilience of our watermark against potential attacks during dissemination, we undertook comprehensive robustness experiments. It is essential to emphasize that our DMs, alongside our watermark encoder and decoder, are proprietary technologies and remain confidential. Consequently, attacks are constrained to execute under a black-box assumption, focusing primarily on post-processing manipulations of the content. In light of this, we assessed the robustness of the watermarked audio against various disturbances, including random Gaussian noise, low-pass and band-pass filtering, stretch, cropping, and echo. These specific attacks are elaborately detailed in the Appendix. Table 4 summarizes the results of robustness for Groot and four SOTA methods against individual attacks.The experimental results validate the excellent robustness of the proposed Groot against individual attacks. Groot demonstrates superior robustness against Gaussian noise and echo compared to To delve deeper into the robustness of Groot with increased rigor, we evaluated its performance under five composite attacks, each integrating two distinct types of individual attacks, alongside one composite attack that combines three single attacks. The specific attack combinations are detailed as follows. 1) low-pass filtering succeeded by Gaussian noise, 2) band-pass filtering followed by an echo attack, 3) cropping and subsequently stretching, 4) Gaussian noise followed by echo, 5) Gaussian noise coupled with band-pass filtering and, 6) Gaussian noise succeeded by band-pass filtering coupled with echo.The results of the robustness experiments against compound attacks are presented in Table 5 . Concretely, Groot illustrates exceptionally superior robustness against compound attacks 1 and 4, with watermark extraction accuracy far surpassing other SOTA methods. Despite facing multifaceted challenges of compound attacks 4 and 5, the recovery accuracy maintains at 99.22% and 99.24%, respectively. Even when confronted with compound attack 6, which amalgamates three separate attacks, Groot sustains an accuracy level of over 90%, peaking at 98.23%. Although DeAR exhibits reasonable robustness against compound attack 2, it fails to maintain the anticipated robustness against compound attacks 1 and 3. In contrast, Groot consistently delivers commendable accuracy across all composite attacks, underscoring its well-balanced and dependable robustness. Furthermore, Fig. 7 utilizes a radar chart to visually represent the robustness of various SOTA methods in countering compound attacks. A more extensive area within the radar chart signifies enhanced robustness. The radar chart for Groot distinctly reveals a larger coverage area, highlighting its superiority over SOTA methods. 
conclusion: 
 In this paper, we propose Groot, a novel generative audio watermarking method, aimed at effectively addressing the challenge of proactively regulating the generated audio via DMs. Groot instills the watermark into DMs, enabling the watermarked audio can be generated from the watermark via DMs. Leveraging our designed watermark encoder and decoder, there is no requirement for retraining the DMs. Robustness has been enhanced precisely due to the meticulous watermark decoder and jointly optimized strategy. The experimental results and comparisons to SOTA methods further illustrate that Groot exhibits potent and well-balanced robustness capable of countering individual and even compound attacks while ensuring superior fidelity and capacity. Regarding future work, we will introduce the noise layer and incorporate the more suitable watermark encoder and decoder to boost the fidelity and robustness of the generative watermarking method. 
 
 Appendix:  


Decision: This paper presents Groot, which can serve as a plug-in to audio diffusion models to add robust watermarks. The reviewers find the approach achieves outstanding performance over prior work. There are some weaknesses that should be stated clearly. For example, the watermark bits are fixed during inference. Also, clarify how cropping is performed. The AC recommends acceptance of the paper and suggests the authors to revise the paper accordingly.

Reviews: review: paper_summary: paper_summary: In this paper, the authors proposed a generative robust audio watermarking method to proactively supervise the synthesized audio and its source diffusion models. The idea is to design the lightweight encoder/decoder for watermarking. The idea is novel and the manuscript is well written.
strength_weakness: strength_weakness: The proposed method achieves a remarkable accuracy gain over the SOTA methods. Besides, the algorithm is described concisely and clearly with a good reproducibility.
limitations: limitations: The sigificance of using watermark for the generative audio synthesis should be further investigated. Is it used for copyright protection of the artificial content? It seems that the generative models with/without watermarking modul may yield the similar content. How to distinguish the ownership of the generative content derived from the other legal model and the proposed method?
paper_summary: paper_summary: This paper proposes a robust generative audio watermarking method coined Groot, which synchronizes the process of watermark generation and audio synthesis. Groot incorporates a dedicated watermark encoder, a parameter-fixed diffusion model, and a lightweight encoder. Experimental results validate that Groot exhibits remarkable performance in terms of watermark robustness, capacity, and audio quality.
strength_weakness: strength_weakness: The implementation of Groot can be a plug-and-play solution for any existing diffusion model. The training overhead is acceptable since it does not require retraining the diffusion model. It is a promising generative audio watermarking method tailored for diffusion models. The evaluation is also thorough, including robustness tests on various individual and compound attacks.
limitations: limitations: Although the proposed generative watermarking framework has no noise layer, it is surprisingly robust and can resist various common distortion attacks. However, I am concerned about whether the framework can withstand desynchronization attacks, such as pitch shift, time stretch, and jitter.

Some experimental results necessitate more detailed explanations. For example, the presence of negative values in Table 3, and the trade-off between audio quality and extraction accuracy at high capacities, as shown in Figure 5.

In the inference stage, it is unclear whether the embedded watermark bits can be changed flexibly. Moreover, I would like to know if the proposed watermarking paradigm can adapt to generative image watermarking schemes, even though they involve different media modalities.
paper_summary: paper_summary: A new method was proposed for embedding watermark information into the generated audio.
strength_weakness: strength_weakness: The embedding stage is merged with the generation stage. According to the special designed neural network structure, the comprehensive performance of the proposed watermarking method is improved. This method is based on the typical codec structure, and has certain innovation on embedding the watermark into the generated audio. The experimental setup is reasonable and the experiments are sufficient. The manuscript was well-written and organized, which is easy to follow for the readers.
limitations: limitations: 1. It is necessary to clarify the respective roles played by the watermark embedding mode (generative watermarking), the encoder and decoder in the overall performance improvement, and better reflect the advantages of embedding watermark in the generation stage. 
2. The further explanation is needed to clarify why the attack layer was not added during the training, but still achieved good results in the attack experiments.
3. In the last paragraph of Sec. 4.3, whether the subscript "Audio" of the loss function value refers to "Aud".
4. It seems that there are no experimental results of proposed Groot based on DiffWave in Table 2.
5. As shown in Figure 2, the noised latent is inputted into LDM during training, and the latent σ is inputted into LDM during inference. Why is the input for the inference process not noised latent? Can the distributions of the noised latent and the latent σ remain consistent?
6. The results of comparison experiment can only demonstrate the performance of the proposed method, the authors are suggested to show the relationship between the performance improvement and watermark embedding before audio generation. In other words, how does the watermark embedding before audio generation work on the performance improvement should be clearly justified and demonstrated.
paper_summary: paper_summary: In this paper, a watermarking method for vocoder based on diffusion model is proposed. Specifically, the watermark is embedded by adding the watermark feature to the random initial latent code of diffusion vocoder before the waveform generation process, and the watermark is extracted from the generated waveform by the corresponding decoder. The identity of the model is verified by the extracted watermark. In this example, the process of watermark embedding and wavform generation occurs at the same time, and in the experiment, the proposed method outperforms the leading state-of-the-art post-processing based watermarking method.
strength_weakness: strength_weakness: - Innovatively introduce generative watermarking into the field of audio watermarking, and add watermarking to the audio vocoder. In recent years, researchers have conducted extensive research on model watermarking, but most of them used image generation model as an example to demonstrate the performance of the method, without exploring it in the audio scene. This paper attempts to explore generation model watermarking the audio field.
- The logic is clear and the paper is written in an easy-to-understand manner.
- The paper conducted a large number of experiments to explore the effectiveness of the method.
limitations: limitations: - Insufficient motivation description. Why is vocoder watermarking needed? It needs to be explained clearly, if the post-processing based audio watermarking is enough.
- Insufficient contribution. The proposed method seems to be a special case of diffusion model watermarking (like Tree-Ring[1]), just an application in the audio vocoder scenario.
- Appropriate methods are not cited (Tree-Ring and other general diffusion watermarking). As a general diffusion watermark, the methods can be directly used in diffusion-based vocoders (a special case) as a baseline.
- How is the Cropping test implemented? Through supplementary, it is found that the length of the audio will change due to cropping. According to the description of the network structure of the proposed method, the input and output sizes of the decoder should be fixed. So after cropping, how to extract the watermark? 
- Lack of generality testing, no cross-testing of the watermark extractor between different vocoders, ie. traing on A and test on B, "plug-and-play" does not make sense.
- The regeneration process will cause great losses, the regenerated audio maybe very different from the original audio, which is bad for users. Why the "D-" indicator is used to measure audio quality. This seems unfair to the compared post-processing based watermarking methods. Moreover, the SSIM index of audio containing watermark will be strangely higher than that of audio without watermark, which shows that it is unreasonable to use "D-" (difference values) as an index.
- How does the proposed watermarking algorithm solve the problem of desynchronization? WavMark obtains desynchronization robustness through shift distortion in the noise layer to face distortions such as cropping and I found that TimbreWM mainly embedding watermark information in frequency to face this problem. I would like to revise the rating upon addressing this question.

[1] Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust