VoiceBlock: Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models
Authors: Patrick O'Reilly
Andreas Bugler
Keshav Bhandari
Max Morrison
Bryan Pardo
Venue: NeurIPS 2022 Accept (2022)
Keywords: privacy
adversarial examples
speech
speaker recognition

Abstract: As governments and corporations adopt deep learning systems to collect and analyze user-generated audio data, concerns about security and privacy naturally emerge in areas such as automatic speaker recognition. While audio adversarial examples offer one route to mislead or evade these invasive systems, they are typically crafted through time-intensive offline optimization, limiting their usefulness in streaming contexts. Inspired by architectures for audio-to-audio tasks such as denoising and speech enhancement, we propose a neural network model capable of adversarially modifying a user's audio stream in real-time. Our model learns to apply a time-varying finite impulse response (FIR) filter to outgoing audio, allowing for effective and inconspicuous perturbations on a small fixed delay suitable for streaming tasks. We demonstrate our model is highly effective at de-identifying user speech from speaker recognition and able to transfer to an unseen recognition system. We conduct a perceptual study and find that our method produces perturbations significantly less perceptible than baseline anonymization methods, when controlling for effectiveness. Finally, we provide an implementation of our model capable of running in real-time on a single CPU thread. Audio examples and code can be found at https://interactiveaudiolab.github.io/project/voiceblock.html.

VoiceBlock: Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models
Patrick O'Reilly, Andreas Bugler, Keshav Bhandari, Max Morrison, Bryan Pardo
Department of Computer Science, Northwestern University
(patrick. ore1ll2024,, andreas, keshavbbardar12223)60
northwestern. serrrereee..............reee.ee. edu pardo@northwesteern. edu
Abstract
As governments and corporations adopt deep learning systems collect and analyze user-generated audio data, concerns about security and privacy natu- rally emerge in areas such as automatic speaker recognition. While audio ad- versarial examples offer one route to mislead or evade these invasive systems, they are typically crafted through time-intensive offline optimization, limiting their usefulness in streaming contexts. Inspired by architectures for audio-to- audio tasks such as denoising and speech enhancement, we propose neural network model capable of adversarially modifying a user's audio stream in real- time. Our model learns to apply a time-varying finite impulse response (FIR) filter to outgoing audio, allowing for effective and inconspicuous perturbations on a small fixed delay suitable for streaming tasks. We demonstrate our model is highly effective at de-identifying user speech from speaker recognition and able to transfer to an unseen recognition system. We conduct a perceptual study and find that method produces perturbations significantly less per- ceptible than baseline anonymization methods, when controlling for effective- ness. Finally, we provide an implementation of our model capable of running in real-time on a single CPU thread. Audio examples and code can be found at httpes//inneerattivvadbllll gtt.ub.io//rojjett/oooieeeeoooee
html
Introduction
Mass surveillance voice communications is an ongoing and pervasive issue. While section 702 of the United States Foreign Intelligence Surveillance Act allows the government perform targeted monitoring of foreign communications, bulk collection practices have resulted in the warrantless surveillance of large numbers of "incidental" foreign and domestic individuals (17). Despite the fact that millions of these communications are obtained without warrants, they have been used in ordinary criminal investigations [60). undermining core purpose of the Fourth Amendment of the US constitution [59]: to protect the people against searches without probable cause. Many corporations also possess the capability for large-scale collection of voice data 233 which may be leveraged to profile users for advertising or accessed by government entities through upstream and downstream surveillance [6 As individuals are faced with these growing surveillance apparati, worth remembering that routine surveillance of private voice communications is also corrosive to free speech and association and tends to disproportionately affect marginalized groups [28] 18).
In the absence of identifying metadata, automatic speaker recognition systems can facilitate mass surveillance by allowing operator search a database of recorded voice data for utterances from chosen speaker 7 or to diarise (assign utterances to individuals) transcripts of recordings [4]. Prior the advent of automatic speaker recognition these tasks required human analysts, forming a natural check on surveillance overreach. We seek to restore this check by degrading the efficacy of speaker
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

recognition models while maintaining the original perceptual quality of the voice communication, step that could grant users measure of privacy from mass surveillance.
Modern speaker recognition systems rely on deep neural networks [6]] Deep networks have been shown be vulnerable adversarial examples- natural instances (e.g- recording of "Bob" speaking) modified to cause model to make an incorrect prediction (the recording's 's speaker is labeled as "Maria") [53). This presents opportunities for privacy-minded individuals mislead or evade surveillance systems with adversarially-crafies inputs. Researchers have proposed adversarial
attacks against a variety of audio systems, including speaker recognition 62
To fool a given neural network-based system, many audio attacks modify a recording by adding perturbation signal directly the waveform representation ( This perturbation is typically crafted using gradient information obtained from the system's neural network-or from a similarly- constructed surrogate-ann requires iterative optimization. Once the optimization is complete, the modified recording is played to the system in an effort to induce arbitrary incorrect predictions (an untargeted attack) or a specific incorrect prediction chosen by the attacker (a targeted attack).
In theory, such attacks might allow individuals to evade systems that surveil and analyze voice communications, thereby protecting privacy. However, many existing attack algorithms require costly optimization for each audio recording to which they are applied. Continuous, real-time voice communication precludes the adoption of such approaches through online optimization (which is typically too slow) or through the use of a set of precomputed adversarial examples (which would necessarily limit the user's interaction). This suggests need for algorithms capable of modifying speech on-the-fly.
Recent years have seen the development of models for audio-to-auee tasks such denoising, voice conversion, and musical timbre transfer [33 48 [4]. Such models have sufficient expressive power to modify audio in coherent and task-specific ways, and are often designed to run in real-time. Inspired by these models, we propose VoiceBlock, deep network that learns to apply a time-varying finite impulse response (FIR) filter outgoing audio, producing highly effective adversarial perturbations on a small fixed delay suitable for streaming. The main contributions this work are:
A highly effective method for de- de-identifying speech in real-time that is more perceptually inconspicuous than existing methods of equal effectiveness
Objective and subjective experimental results that validate these claims
A system that embodies the claimed advances, and that runs in real-time on a single CPU thread
Through this work, we hope to encourage further exploration of audio-to-audio models for protecting user privacy, Audio examples and code, including our streaming implementation, can be found at https:/interactiivadd..e
wttthu...o/roojecc//oooooooooo ttttp..../pppppee//oollelll..... html
2 Related work
Previous works in speech de-identification have proposed methods for obfuscating speech to evade surveilling ppeaker-recognitin
systems. One such method is voice conversion, which modifies the speech of a source speaker to sound like that of another target speaker, both to humans and machines. Jin et al. [24] and Alegre et al. [4] analyze the vulnerability of speaker recognition systems against de-identification attacks performed with voice conversion models. However, the conversion models considered are incapable of real-time operation. While the recent availability of low-latency voice conversion models [48] 49 31] may make such approaches more practical, our aim is to modify speech in way that is inconspicuous and minimally invasive to the user experience, and that has the potential to generalize to tasks beyond speaker recognition. This also rules out recent anonymization approaches catalogued through the VoicePrivacy Challenge, all evaluated submissions to date noticeably alter speaker characteristics [35]. Similarly, we do not consider obvious transformations like pitch-shifting that significantly alter degrade recorded audio.
Real-time audio attacks have recently been explored. Chiquier et al. 10 propose an attack on automatic speech recognition systems that is capable of inducing significant transcription errors in an over-the-air environment (where attack audio played to the system a physical space), but which introduces conspicuous noise due to the use of additive perturbations at the audio waveform. Their

---

approach also necessitates both a large model (requiring approximately 16 GPU-days to train) and lengthy receptive field, resulting an initial "idle" period of 2.5s during which the attack does not affect speech. By contrast, we focus on an over-the-line setting, where attacks are passed to the victim model over a digital channel, and propose a lightweight network capable of producing inconspicuous perturbations through the use of filtering, with an initial delay of miliseconds rather than seconds.
Universal adversarial attacks optimize short perturbation that can be played alongside arbitrary speech, optionally in real-time, to evade a victim model [2 62 27 However, these attacks tend to introduce conspicuous noise due the use of additive perturbations the audio waveform. The same holds for the attack of Xie et al. [6]], which uses a Wave- N-N [52] model to efficiently generate universal perturbations. Rather than generate additive perturbations, we perform multiplicatiie attacks in the frequency domain through the use of filtering. and in doing so avoid the bias towards noise-like artifacts typical of additive attacks.
Other works have proposed crafting adversarial attacks in the frequency domain. The "Ke- nansville" attack of Abdullah et al. M1 performs spectral gating, removing low-energy frequency components in an effort to hinder classification. The authors demonstrate the effectiveness of their approach against seen and unseen speaker- and speech-reconnition systems, and the proposed method is fast and gradient-free. However, the spectral gating must be constrained to a small number of key audio frames avoid conspicuous artifacts, which requires offline optimization using word- or phoneme-leve alignment information.
Ahmed et al. optimize bandpass filters to perform targeted impersonation attacks on speaker- recognition models; these filters can then be physically realized as resonant tubes through which an attacker speak to the victim system, allowing real-time operation. However, once optimized and realized, the bandpass filters are fixed in time and cannot adapt to the attacker's 's speech. The attack is significantly less effective than traditional approaches, even in controlled acoustic environments, and requires an active intervention on the part of the user to both realize and apply.
Finally, O'Reilly et al. [41] adversarially optimize the parameters of time- varying finite impulse response (FIR) filter in order to avoid noisy artifacts. However, the attack operates offline and must be optimized separately for every instance to which it is applied.
In summary, we are aware of no current approach that simultaneously capable of performing de- identification while remaining inconspicuous to human listeners and operating in real-time on-device.
3 VoiceBlock
We propose VoiceBlock, a system that applies adversarial time-varying filtering to user audio in real-time. VoiceBlock modifies speech by specifying the frequency magnitude response of standard finite impulse response (FIR) filter. By carefully varying the ilttr's response many times per second, the speech is de-identified to an automated system while speaker identity is preserved for human listeners. This is accomplished without introducing noisy artifacts at the waveform, as filtering can only amplify or attenuate frequency energy present in the original signal through multiplication the Fourier domain (see Section 3.3 for details of the filtering implementation used in our experiments).
VoiceBlock consists of three main modules, shown in Figure T (1) an encoder module extracts acoustic features from input audio frames, (2) recurrent bottleneck module incorporates context from past frames predict set of adversarial filter controls for each frame, and (3) decoder module regularizes the predicted filter controls and applies them to the corresponding frame of input audio. We provide an overview of each component below, and further details in Appendix
3.1 Encoder
The encoder module extracts acoustic features from 16kHz audio (double the sample rate of telephone speech) segmented into frames of 256 samples with 50% overlap. Though our model can operate in fully causal fashion, we find we achieve stronger attacks using a lookahead of five frames (see Section
3.2). resulting in a theoretical latency of 56ms. For reference, Verizon reports 25ms or less is typical latency for voice-over-IP communication in North America and Europe [56), while experimental data
266 show that 130ms of latency is acceptable for high-quality voice communications, Therefore,
adding VoiceBlock to the signal chain leaves latency well within this limit.
3

---

Embedding space of recognition system
Encoder
Decodar
Spenicer utterance
Speaker utterance
Query
Bottlenock
VoiceBox
VoiceBlock
IWE
SunpoN Back.
Figure 1: Left: The proposed VoiceBlock architecture. Acoustic features extracted by the encoder are fed the recurrent bottleneck to predict filtering controls, which are regularized and applied to the input by the decoder to obtain adversarial audio. Right: VoiceBlock adversarially perturbs the user's audio stream such that any extracted queries are scored by the system as dissimilar to the user's enrolled utterances, hampering identification or retrieval.
Audio frames entering the encoder are passed four sub-modules in parallel to obtain the following features.
Pitch features: Given the known sensitivity of speaker-recognitin models pitch and the importance of spectral structure in delincating linguistic content, we extract fundamental frequency and aperiodicity estimates for each frame using the DIO algorithm (36). Pitch estimates are refined by the StoneMask algorithm [37] For both stages, we use the pyworld 22 implementation of the
WORLD vocoder [37] (MIT license).
Loudness: We take an A-weighted average of each frame's log-magnitude spectrum to obtain a loudness estimate (35)
Following the method of Ronssin Cernak 48 we use trained phoneme classifier with frozen weights to encode linguistic content. The classifier consists of multi-layer perceptron followed by two LSTM (21) layers and linear classification layer, and takes as input 13 mel-frequency cepstral coefficients (MFCC) with first- and second-order deltas for each frame. We train our phoneme classifier on the "train-clean- 100" subset of the LibriSpeech dataset using frame-aligned phoneme labels 33 34 (CC-BY 4.0 license). Rather than directly the classifier's predicted distributions over phoneme labels known as phonetic posteriorgrams (PPGs)
Phonetic posteriorgrams:
42
19 we discard the classification layer after training and pass along the output of the final LSTM layer, which Ronssin & Cernak also refer to as PPGs in their architecture.
Spectrogram features: Finally, we use a simple network consisting of gated linear unit and multi-layer perceptron operating independently on each mel-spectrogram frame-a a widely used speech representation [25 to capture any residual information. We note that Engel et al. [14] also
make use of a spectrogram encoder, alongside pitch and loudness features, to control differentiable signal-processing components.
For each frame, the above features are concatenated and projected linearly to obtain a low-dimensional encoding. We then introduce additional speaker information by passing fixed, pre-computed embedding of the source speaker through a FiLM layer (45) to modulate the encoder output. This embedding helps to guide the de-identification task, allowing to train single model capable of de-identifying arbitrary users. This imposes a requirement that users must record a small amount of speech to obtain embedding before first using VoiceBlock. We find that less than minute of

---

speech is sufficient, and use a pre-trained ResNetSE34V2 model [20 to compute embeddings (see sections
3.2 Bottleneck
Encodings entering the bottleneck are passed through two LSTM layers, and the outputs are con- catenated with skip connection from the encoder. enable streaming, we use unidirectional LSTM layers and pass concatenated outputs through a small lookahead convolutional network [57] to incorporate information from future frames the expense of a small fixed delay. We find lookahead of 5 frames (48 ms) is sufficient to craft strong de-identification attacks. Finally, linear projection layer maps the concatenated representations to a vector of filter controls, with each element representing the unnormalized frequency magnitude response of filter band for the current frame.
3.3 Decoder
Our decoder applies time-varying filtering to the input audio based on trame-wise controls obtained from the bottleneck. For our task, we find that tntering-baset decoder affords number of advantages over the synthesis-band based (e.g. cranspose-coonvolutiona decoder architectures present in many audio-to-audio models [13]
The filtering module is not prone to the periodic upsampling artifacts that transposed- convolutional architectures often introduce [46].
Our decoder is simple deterministic module with trainable parameters, keeping Voice-
Block lightweight.
The capacity of the decoder and conspicuousness of perturbations can easily be constrained in terms of the number of filter bands their allowed range of motion, providing inter- pretable control to the user.
To regularize and apply filter controls each frame of audio, we use a method similar that of O' O'Reilly et al. and Engel et al. [41 [4]. We apply sigmoid scaling to bound filter controls to the range [0, and clip deviations from unity beyond a fixed value Each set of scaled filter controls is transformed into time-domain impulse response via the inverse Fourier transform. We shift the impulse response to zero-phase (symmetric) form, apply a Hann window, and finally convolve with the corresponding input audio frame by taking the Fourier transform and performing element-wise multiplication. After compensating for shift, we overlap-add the resulting frames with Hann window to obtain the final filtered audio. We discuss the details of our buffered streaming implementation in Appendix A.3 Our VoiceBlock model is implemented in PyTorch 2 and contains 6.3m trainable parameters, and 7.5m in total, counting the frozen phoneme encoder.
3.4 Training objective
To demonstrate the ability VoiceBlock to perform inconspicuous privacy-preserve audio trans- formations, train model to attack speaker recognition systems. Given large database of speech recordings and access to a user's 's audio stream, a surveilling entity may seek to (a) identify the user by matching their speech against recordings of known provenance in the database, (b) retrieve other utterances of the user from the database. Systems designed for these tasks - speaker recognition and retrieval, respectively often rely on neural network models map speech utterances to low-dimensional embedding space in which distance corresponds to speaker similarity [15]. In this work, we consider models for which the speaker distance D, between utterances and can be measured via a cosine distance between embeddings f(u) and f(v):
f(u) f(o)
() =1 [If())ll llf(v))22
(1)
At inference time, query audio from the user is embedded and scored for similarity against enrolled utterances pre-computed embeddings stored the database. Scores may be evaluated for all enrolled embeddings, or for only representative of each unique speaker (e.g. speaker centroids). The highest-ranking result or results (e.g. the closest speaker identity) are then returned. We do not distinguish between recognition and retrieval tasks, and refer to both under the umbrella of "speaker
5

---

recognition." This is because from privacy standpoint, the objective in each task is identical: alter the query audio prevent valid matches, thereby de-identifying the user.
variety of methods have been proposed to efficiently compute and search over low-dimensional speaker representations. Generally, hashing algorithm is applied to speaker embedding vectors to reduce storage and search costs [50 30 [5]. Because the resulting hash representation merely serves as an efficient point of access for embedding-space distances, we omit hashing algorithms from consideration and define our attack objectives on the embedding space directly.
Given speaker embedding model, we aim modify query audio drawn from a user's stream such that its embedding- -pacc distance from any enrolled utterances of the user is large, and its evaluated similarity is small. We quantify this de-identification in terms of distance thresholds in the embedding space, set according to percentiles the estimated distribution of all inter-speaker embedding distances. Let P. represent the distance corresponding to the y percentile of this distribution; then for query audio u. and enrolled utterance v. (u, v) >> P, implies that roughly r percent of database entries should be scored as more similar than v. Thus, we can set distance thresholds that correspond directly to the strength of de-identification applied to user audio, and cosine distance between
construct loss function that penalizes our model when the embedding-spea query and enrolled utterances falls below the threshold. To do so, we use a variant of the adversarial loss proposed by Zhang et al. 662. Let f represent the victim model, 9 our VoiceBlock network, an utterance from our user's 's audio stream, and P. our de- de-identification threshold; then
Ladv(S u) - (P, - (g((), u) + +***
where (+)+ max(-,0) 0) and k is confidence parameter encouraging the attack fully cross the threshold. To ensure that our VoiceBlock model learns to perturb user audio inconspicuously. we incorporate additional loss function to penalize perceptible filtering artifacts. We compute the combined waveform L, and multi-resolution spectrogram losses proposed by Defosséz et al. [13 on the clean and adversarial audio:
Lauz(g, u) = (u, g(u)) + 1|u-~ll
- g(u)l1
(3)
where Lstft is given by sum of magnitude and spectral convergence losses computed over sev- eral spectrogram resolutions. We provide further details in Appendix A.. Combining the above adversarial and auxiliary losses, we obtain our final attack objective:
= Lado Laux
(4)
4 Experimental design
We describe experiments used to validate the claimed advances of our work, namely that VoiceBlock can de-identify speech in real-time while remaining significantly less conspicuous than existing methods of similar effectiveness. We first introduce the models, datasets, and attacks considered in our experiments. Following this, we detail our experiment configurations and present the results of both objective and subjective evaluations.
4.1 Speaker recognition models
ResNeeSE34v22: We train attacks against the ResNetSE34v2 model 20 provided in the VoxCeleb Trainer repository [12] (MIT License). The model takes mel-spectrogram inputs and uses 2D convolutions with residual connections, squeeze-and-exeitati and attentive statistics pool- ing generate frame-level features and aggregate them into 512-dimensional speaker embeddings.
Y-Vector: To examine the transferability of our approach against unseen systems, we evaluate trained attacks against the Y-Vector model proposed by Zhu et al. [63]. The model uses multiscale ID-convolutional waveform encoder to extract acoustic features, followed by weareeezeanddexxiittteee blocks, feature aggregation, and a time-delayed neural network map variable-length utterances to 128-dimensional speaker embeddings.
Both the ResNetSE34v2 and Y- Vector models were trained on the development set of the VoxCeleb2 dataset [I]. Note that while our attack is mousal- a short, five-frame lookahead-ov do not impose the same restriction on a hypothetical surveillance system; instead, we evaluate our attack against strong, non-causal systems capable of aggregating speaker characteristics from across full utterances before rendering predictions.

---

4.2 Datasets
LibriSpeech: (CC-BY 4.0) We use both the train-clean-100 and test-clean subsets of the LibriSpeech dataset [42] for training VoiceBlock. The former comprises 28,539 utterances from 251 speakers while the latter comprises 2,620 utterances from 40 speakers.
VoxCelebl: (CC-BY 4.0) To simulate large-scale surveiling speaker recognition, we evaluate attacks on the VoxCeleb1 dataset (39], comprising 153,516 utterances from 1,251 speakers. This also ensures that no evaluation speakers are seen during training. As with the LibriSpeech dataset, we trim or pad all utterances seconds.
For all experiments, we carefully divide the data to imitate a realistic attack setting. During training, we select fifteen utterances (one minute total) from each source speaker in the training set and compute embeddings using the ResNetSE34v2 (MIT License) model. The centroid of these embeddings is then used as enrolled target for the computation of the adversarial loss with all utterances of that speaker. We find that this produces stronger attacks than using individual utterance embeddings as targets, possibly by ensuring more consistent gradient information across the optimization. For our VoiceBlock attack (see Section 4.3). we select a further ten utterances (40s total) from each source speaker in the training set and again compute embeddings using the ResNetSE34v2 model. The centroid of these embeddings is then fed as a fixed conditioning vector to the VoiceBlock model alongside all utterances from that speaker (see Section 3.1). Similar training, during evaluation we select fifteen utterances per speaker as a query set. We again select a further ten utterances to serve as conditioning for the VoiceBlock attack. Finally, twenty utterances of each speaker are enrolled in the speaker recognition system, serving as the database against which query utterances are matched.
4.3 Attack algorithms
We perform untargeted attacks using the following algorithms. VoiceBlock: We implement the proposed VoiceBlock attack as described in Section Appendix A and train for 10 epochs on 3 NVidia RTX 2080 Ti GPUs; this takes approximately 40 minutes. Universal: We optimize short (2s) universal additive perturbation for 10 epochs using the established penalty method 33 62 27 and the same adversarial objective as VoiceBlock. On 3 NVidia RTX 2080 Ti GPUs. this takes approximately 16 minutes. To limit the perceptibility of the attack, we scale the perturbation to have Loo norm 0.08 times that of the unperturbed speech. During training and evaluation, the perturbation is aligned arbitrarily with query utterances and looped to match durations, serving as a constant adversarial "background" signal. White noise: We add Gaussian noise to utterances at the waveform representation at bignll-t-noiss ratio of 10dB. Spectral gating: We modify the "Kenansville" attack of Abdullah et al. [ to allow for streaming use by performing spectral gating at all frames, using a threshold of 4dB relative to the maximum-energy spectral bin of each frame.
4.4 Objective evaluation
We evaluate attacks in large-scale closed-set speaker recognition task. First, we use the test-clean LibriSpeech subset to obtain a rough estimate of the distribution of distances between an individual utterance and the centroids of each distinct speaker in the embedding space of the ResNetSE3442
model. To encourage strong de-identification attacks, we take the distance corresponding to the 25th percentile of this distribution as the target threshold Pıs for our training loss (see Section 3.4 We train each attack as discussed above.
We evaluate the closed-set recognition of all attacks over the VoxCelebl dataset. We compute the distance between each query embedding and the centroid of all embeddings of each speaker; recogntion is then performed by returning the speaker identity of the nearest speaker centroid. We find this is slightly more accurate and robust to attack than using the identity of the nearest embedded utterance. We perform exact nearest-neighbors search over the embedding space, and report the top-1 (T-1) and top-10 10 (T-10) accuracy of the relevant speaker recognition model given both clean and adversarial queries. Additionally, we compute the following objective speech quality metries over the clean and adversarial query audio as a proxy measure of the imperceptibility of attacks: Perceptual Evaluation of Speech Quality (PESQ) [47), operating in the wide-band configuration and using the python-pest -pesq implementation 58 (MIT license); and Short-Time Objective Intelligibility (STOI) [54], using the pystoi implementation (MIT license). The results of our evaluation are presented in [
7

---

Table 1:
Results of our objective evaluation. We perform attacks on the VoxCelebl dataset against seen (ResNetSe34v22) and unseen (Y- (Y-Vector) speaker recognition models, and compute both top-1 and top-10 10 recognition accuracies. Additionally, we compute a set of objective speech quality metrics for each attack.
Speech Quality Metrics
ResNetSe34V2
V-Vector
Approach
PESQ f
STOr f
T-1
T-104
T-1
T-10
White noise
1.03
0,41
0.16
0.47
0.00
0.01
Spectral gating
1.11
0.63
0.02
0.12
0.03
0.13
Universal
1.35
0.78
0.09
0.20
0.48
0.73
VoiceBlock
3.74
0.92
0.03
0.10
0.35
0.66
No attack
4.64
0.99
0.99
0.99
0.96
0.99
We evaluate the real-time performance of the streaming implementation of the VoiceBlock attack described Appendix by measuring its average real-time factor (RTF). measure performance on single thread on two different CPUs, an Intel i7-5600U @ 3.2 GHz and an Apple M1 Chip. In the streaming configuration, VoiceBlock processes a chunk of 4 overlapping frames
time, equivalent to 640 samples / 40 mss We compute the average RTF for processing chunk over all chunks in a 4 second audio clip, and find that VoiceBlock has an average RTF of .255 on the Intel 17-5600U and .200 on the Apple MI.
4.5 Subjective evaluation
For our subjective evaluation, we sampled 100 clean speech recordings from the VoxCelebl dataset. To standardize comparisons, we trimmed or padded utterances seconds. Each of the four attacks described in Section 4.3 was applied to all 100 recordings, resulting in 100 comparison sets with five recordings per set: the original clean speech and the speech modified by each of the four attacks. These sets were then evaluated in MUSHRA-style listening study [9] deployed on Amazon Mechanical Turk using the open-source, MIT-licensed Reproducible Subjective Evaluation (ReSEval) (38 system. IRB approval was obtained prior to conducting this study, and there were no known risks to the participants in this study.
100
80
60
40
20
noise
universal
spectral gating
voicebox
clean
Figure 2: Distributions of quality ratings from our crowdsourced subjective listening MUSHRA-style test on audio quality. Higher numbers are better. Black dots are means and white dots are medians. Wilcoxon signed-ranked tests between all pairs of conditions show statistical significance < 0.05.
We recruited 20 participants. Participants were screened with listening test prior to beginning the study; those that passed rated 20 comparison sets. For each comparison set, the participant was asked to listen to and rate the relative audio quality of each of the five audio files on scale from 0 to 100. We omitted responses by four participants who failed our prescreening listening test and nine who rated the white noise attack (our low anchor) as superior in quality to ground-truth clean audio (our high anchor), giving us a total of 140 five-way comparisons. For more details of our crowdsourced subjective evaluation, see Appendix C

---

The results of our crowdsourced subjective evaluation can be found in Figure 2 preferred to all other methods, and exhibits similar perceptual quality We find VoiceBlock sround-truth speech recordings. The difference between VoiceBlock and ground-truth audio is significant (p < 0.05) using a Wilcoxon signed-rank test (p = 0.024) but not significant using Welch's T-test (p 0.071), which assumes that human perceptual scores are normally distributed but with potentially different variances, All other pairs of conditions are significantly different using either test.
4.6 Additional experiments
In Appendix B. we conduct ablation studies on the VoiceBlock encoder configuration, lookahead length, and choice of auxiliary loss. We also conduct supplementary experiments demonstrating the robustness of VoiceBlock against a deep network-bar -hasorr speech enhancement model, examining its performance under the assumption that adversarial queries are enrolled by the surveilling speaker recognition system, and evaluating the intelligibiliiy of processed audio using an automatic system. These experiments are detailed in appendices] B.2 B.3 and B. BndB..4esppeee respecttivvll
Ethics
The main goal of our work is to show that large-scale, untargeted automated surveillance can be hindered by introducing inconspicuous perturbations to audio real-time. Because VoiceBlock does not conceal speaker identity from human listeners, it still allows high-effort targeted surveillance (e.g- authorized suman-attended wiretaps of criminal enterprises). In this way, we hope to return to the status quo of the 20th and early 21st centuries in which the need for human listeners provided an important check on mass surveillance.
Those wishing to avoid all voice identification by human or machine may use existing voice conversion technologies. Voice conversion not, however, appropriate many contexts where some measure of privacy from mass surveillance may be desired. Consider telchealth: a medical expert or therapist discussing sensitive topics with a patient may gain great insight from hearing the nuance of the patient's ' voice. In this setting, methods that alter one's voice beyond human recognition (e.g. voice conversion) would necessarily degrade the interaction.
The approach of VoiceBlock could also, in concept, be used to fraudulenty access information or services protected by speaker verification. However, VoiceBlock performs perceptually inconspicuous filtering and the speech produced would still sound like the original speaker. As such, it could not pass human inspection. Both voice-conversion and speech synthesis-based attacks are much more suited to this purpose, as both produce speech in voice perceptually similar to the target speaker.
Our approach can be thought of leveraging the asymmetry between system and attacker attention inherent to mass surveillance, using inconspicuous perturbations to evade large-scale systems that must render predictions in bulk. This same asymmetry is not necessarily present when trying to bypass authentication mechanisms and access rightly-gurrder services.
6 Conclusions
Our experimental results indicate VoiceBlock can de-identify speech from arbitrary unseen users in real-time on a standard MI CPU. The de-identified speech is of significantly higher audio quality than competing methods, as reported by subjective listener study and as measured through standard metrics of speech intelligibility. Our method also achieves nontrivial de-identification results against a system it was not trained on, outperforming a far more perceptible universal attack. This is notable given that we take no explicit steps to improve the transferability of our method, which leverages a lightweight model trained on a small corpus of clean speech without augmentation. While pure approaches such as spectral gating and white noise transfer between systems more successfully, they severely degrade audio quality, resulting in much lower listener quality ratings. This limits their practicality in real-worke voice communications. By contrast, method produces adversarial audio that is virtually indistinguishable from the clean source, as indicated by our subjective evaluation.
We view our method as an initial step towards protecting users from indiscriminate mass surveillance. A number of obvious directions for future work stand out, such as improving the transferability of the

---

adversarial examples crafted by our system [40] and evaluating attacks against real-world speaker recognition systems and over real-worke communication channels. We hope this work encourages further exploration of the applications of audio-to-audio models for protecting user privacy.
References
[1] Hadi Abdullah, Muhammad Rahman, Washington Garcia, Kevin Warren, Anurag Yadav, Tom Shrimpton, and Patrick Traynor. Hear "no evil", see "kenansville"** * Efficient and transferable black-box attacks speech recognition and voice identification systems. In IEEE Symposium on Security and Privacy, 2021.
[2] Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga Alban Desmaison Andreas Kopf Edward Yang Zachary DeVito Martin Raison Alykhan Tejani Sasank Chilamkurthy Benoit Steiner Lu Fang Jun- jie Bai Adam Paszke, Sam Gross and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Neural Information Processing Systems (NeurIPS), 2019.
[3] Shimaa Ahmed, Yash Wani, Ali Shahin Shamsabadi, Mohammad Yaghini, Ilia Shumailov, Nicolas Papernot, and Kassem Fawaz. Pipe overflow: Smashing voice authentication for fun and profit. arXiv preprint 1xXix:2107.14644 2022.
[4] Federico Alegre, Giovanni Soldi, Nicholas Evans, Benoit Fauve, and Jasmin Liu. Evasion and obfuscation in speaker recognition surveillance and forensics. In International Workshop on Biometrics and Forensics, 2014.
[5] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Neural Information Processing Systems (NeurIPS), 2020.
[6] Zhongxin Bai and Xiao-Lei Zhang. Speaker recognition based on deep learningg An overview. Neural Networks, 140:655-99 2021.
[7] Peter J. Barger and Sridha Sridharan. On the performance and use of speaker recognition systems for surveillance. In International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2006.
[8] Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech- to-text. In IEEE Security and Privacy Workshops, 2018.
[9] Mark Cartwright, Bryan Pardo, Gautham J Mysore, and Matt Hoffman. Fast and easy crowd- sourced perceptual audio evaluation. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016.
[10] Mia Chiquier, Chengzhi Mao, and Carl Vondrick. Real-time neural voice camouflage. In International Conference Learning Representations (ICLR), 2022.
[11] Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In Inter- speech, 2018.
[12] Joon Son Chung, Jaesung Huh, Scongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and leksang Han. In defence of metric learning for speaker recognition. Interspeech, 2020.
[13] Alexandre Défosse7, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. In Interspeech, 2020.
[14] Jesse Engel, Lamtharn Hantrakul, Chenjic Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing. In International Conference on Learning Representations (ICLR), 2020.
[15] Lei Fan, Qung-Yuan and retrieval. In Interspeech, 2019.
Yuan Jiang, Ya-Qi Yu, and Wu-Jun Li. Deep hashing for speaker identification
10

---

[16] Electronic Frontier Foundation. Upstream vs. prism. http://www..ff..or/0-5-//////
2018.
[17] Barton Gellman and Ashkan Soltani. Nsa surveillance program reaches 'into the past' to retrieve, replay phone calls. The Washington Post, 2014.
[18] Hannah Giorgis. When the fbi spied on mlk. The Atlantic, 2021.
[19] Timothy Hazen, Wade Shen, and Christopher White. Query-by-exampee spoken term detection using phonetic posteriorgram templates. IEEE Workshop on Automatic Speech Recognition & Understanding, 2009.
[20] Hee Soo Heo, Bong-Jin Lee, Jaesung Huh, and Joon Son Chung. Clova baseline system for the voxceleb speaker recognition challenge 2020. arXiv preprint arXiv:2009.14I1 2020.
[21] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1735:1735 11 1997.
[22]
Hsu. Berchn-Wrrapeer for- for- for--prr--ooo--ooo-oooooooooooo 2021.
pyworld.
http:://gghuub.ccc///eeeeyCCCCCp
[23] Umar lqbal, Pounch Nikkhah Bahrami, Rahmadi Trimananda, Hao Cui, Alexander Gamero- Garrido, Daniel Dubois, David Choffnes, Athina Markopoulou, Franziska Roesner, and Zubair Shafiq. Your echos heard: Tracking, profiling, and ad targeting in the amazon smart speaker ecosystem. arXiv preprint arXiv:2204. 10920, 2022.
[24] Qin Jin, Arthur R. Toth, Tanja Schultz, and Alan W Black. Voice convergin: Speaker de- identification by voice transformation. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Apr. 2009.
[25] Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Cyclegan-vc's Examining and improving cyclegan-ver ves for mel-spectrogram conversion. In Interspeech, 2020.
[26] Murizah Kassim, Ruhani Ab Rahman, Mohamad Azrai A Aziz, Azlina Idris, and Mat Ikram Yusof. Performance analysis of voip over 3g and 4g Ite network. In 2017 International Conference on Electrical, Electronics and System Engineering (ICEESE), pages 37 41. IEEE, 2017.
[27] Andre Kassis and Urs Hengartner. Practical attacks on voice spoofing countermeasures. arXiv preprint urXiv:2107.14642 2021.
(28) Dia Kayyali. The history of surveillance and the black community, https://www. eff org/ deeplinks/2014//// htwp://///////htttotosssrlllll// wwwpraawllaaeeeeaaaaaaa-aaa----- and-bllcc-coommnit 2014.
[29] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
[30] Lantian Li, Chao Xing, Dong Wang, Kaimin Yu, and Thomas Fang Zheng, Binary speaker embedding. In 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP),001 2016.
[31] Yinghao Aaron Li, Ali Zare, and Nima Mesgarani. Starganv2-ve: A diverse, unsupervised, non-parallel framework for natural-sounding voice conversion. In Interspeech, 2021.
[32] Zhuohang Li, Yi Wu, Jian Liu, Yingying Chen, and Bo Yuan. Adypulse: Universal, synchronization-fres and targeted audio adversarial attacks via subsecond perturbations. In CCS, 2020.
[33] Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding. In Interspeech, 2019.
[34] Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: trainable text-speech alignment using kaldi, In Interspeech, 2017.

---

[35] RG McCurdy. Tentative standards for sound level meters. Electrical Engineering, 55(3):260- 263, 1936.
[36] Masanori Morise, Hideki Kawahara, and Haruhiro Katayose. Fast and reliable f0 estimation method based the period extraction of vocal fold vibration of singing voice and speech. Journal of the Audio Engineering Society, February 2009.
[37] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: A rocoder-based
speech synthesis system for real-time applications. IEICE Transactions on Information and Systems, E9.. (199.D(7)1877-11888 2016.
[38] Max Morrison, Brian Tang, Gefei Tan, and Bryan Pardo. Reproducible subjective evaluation.
In ICLR Workshop on ML Evaluation Standards, April 2022.
[39] A. Nagrani, J. Chung, and A. Zisserman. Voxceleb: a large-scale speaker identification dataset. In Interspeech, 2017.
[40] Krishna Kanth Nakka and Mathieu Salzmann. Learning transferable adversarial perturbations. In Neural Information Processing Systems (NeurIPS), 2021.
[41] Patrick O' O'Reilly, Pranjal Awasthi, Aravindan Vijayaraghavan, and Bryan Pardo. Effective and inconspicuous over-the-air adversarial examples with adaptive filtering. In International Conference Acoustics, Speech and Signal Processing (ICASSP), 2022.
[42] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, Librispeech: An asr corpus based on public domain audio books. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.
[43] Manuel Pariente. pystoi. https: //github/bttbb///./ hwt.h//...cmm/prrrettt////ssooop 2021.
[44] Tae Jin Park, Naoyuki Kanda, Dimitrios Dimitriadis, Kyu Han, Shinji Watanabe, and Shrikanth Narayanan. A review of speaker diarization: Recent advances with deep learning. Computer Speech Language, 72, November 2021.
[45] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
[46] Jordi Pons. Santiago Pascual, Giulio Cengarle, and Joan Serrà. Upsampling artifacts in neural audio synthesis. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[47] A.W. W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2001.
[48] Damien Ronssin and Milos Cernak. Ac-ve: Non-parallel low latency phonetic posteriorgrams based voice conversion. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021.
[49] Takaaki Sacki, Yuki Saito, Shinnnosuke Takamichi, and Hiroshi Saruwatari. Real-time, full-band, online dan-based voice conversion system using single cpu. In Interspeech, 2020.
[50] L. Schmidt, M. Sharifi, and I. Lopez-Moreno. Large-scale speaker identification. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.
[51] Ali Shahin Shamsabadi, Francisco Sepúlveda Teixeira, Alberto Abad. Bhiksha Raj. Andrea Cavallaro, and Isabel Trancoso. Foolhd: Fooling speaker identification by highly impercep- tible adversarial disturbances. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[52] Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. In International Society for Music Information Retrieval (ISMIR), 2018.
12

---

[53] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICI.R), 2014.
[54] Cees H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligibility prediction time frequenyyy weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7):2125-21) 2011.
[55] Natalia Tomashenko, Xin Wang, Emmanuel Vincent, Jose Patino, Brij Mohan Lal Srivastava, Paul-Gauthier Noé, Andreas Nautsch, Nicholas Evans, Junichi Yamagishi, Benjamin O' O'Brien, Anaïs Chanclu, Bean-François Bonastre, Massimiliano Todisco, and Mohamed Maouche. The VoicePrivacy 2020 challenge: Results and findings. Computer Speech &amp Language,
74:101362, 2022.
[56] Verizon. Monthly ip latency statistics. http:///www..rrrroocco/////i/sss
2022. [57] Chongshun Wang, Dani Yogatama, Adam Coates, Tony Han, Awni Hannun, and Bo Xiao. Lookahead convolution layer for unidirectional recurrent neural networks. In ICLR Workshop, 2016.
[58] Miao Wang, Christoph Boeddeker, Rafael G. Dantas, and Ananda Scelan. pesq. 2022.
https
//github.come httpa///...m/dddo//////oooooooo/
[59] Human Rights Watch. Q & Us warrantless surveillance under section 702 of the
foreign intelligence surveillance act. -ottrr--tttioo-------o-eeeeeeeee htt.a/aaee............ooooooooee sereeeeeeneeeeeeeelaaeeeee
hrw. http::/wwwww.uuorrrrrraaaaaaa///
9 ana-waaraatless-svvvviiii wwweeillance-
2017.
[60] Human Rights Watch. rantless surveillance. aecret-evideece--nddeeee
Secret
evidence and the threat of more war-
https://sww.tht http://wwwww.r..ooooe/////////// org/ httpg//w///001/01/1111 2018.
[61] Yi Xie, Zhuohang Li, Cong Shi, Jian Liu, Yingying Chen, and Bo Yuan. Enabling fast and universal audio adversarial attack using generative model. In AAAI, 2021.
[62] Weiyi Zhang, Shuning Zhao, Le Liu3, Jianmin Li, Xingliang Cheng. Thomas Fang Zheng, and Xiaolin Hu, Attack on practical speaker verification system using universal adversarial pertur- bations. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[63] Ge Zhu, Fei Jiang, and Zhiyao Duan. Y-vector: Multiscale waveform encoder for speaker embedding. In Interspeech, 2021.
13


Reviews:
[{"review_id": "pj-tIjgrK5", "reviewer": "Reviewer_NLhW", "summary": "This paper proposes a real-time audio conversation to de-identify speakers from a privacy perspective. One of the unique aspects of this model is to consider frequency domain filtering instead of noise addition, which seems to avoid the distortion of speech signals in terms of their speech quality measures (subjective and objective) while maintaining the speaker de-identification functions. This paper significantly contributes to the community by considering that the voice privacy issue has become a severe concern and audio applications require streaming/real-time operations.", "questions": "- Did you perform automatic speech recognition? I'm expecting that ASR would be robust against the \"time-invariant\" FIR filter, and I'm curious how it would degrade the performance due to the proposed method based on the time-variant FIR filter.\n- Is it possible to evaluate this method with the Voiceprivacy challenge benchmark? Then, the paper can compare various techniques proposed in the challenge or literature.\n- \"VoiceBox\" is the same name as the famous (Matlab) speech processing toolkit. https://scholar.google.com/citations?view_op=view_citation&hl=en&user=HEH9cboAAAAJ&citation_for_view=HEH9cboAAAAJ:u-x6o8ySG0sC. Although this is an old toolkit (based on Matlab!), I recommend the authors respect this prior work and avoid using this name.\n- I would like to have more discussions on why the authors use the time-variant FIR filter with more scientific perspectives. I agree that a time-invariant FIR filter would be less distorted than the additive noise as it would just physically change the speaker or room characteristics. But I'm not very sure about the time-variant FIR filter.\n- Can you discuss 56ms? I think this range would be accepted in many speech interface applications. For example, DNS challenges and other real-time speech processing tasks set a similar range for the latency constraint. However, some hearing devices set the latency constraint to less than 10 ms (you can check the Clarity challenge). This kind of discussion makes the paper more attractive since we can imagine the main application of this method.\n- How much is the phoneme classifier part robust against the other languages, speaking styles, and noise environments? Librispeech \"train-clean-100\" is English/clean/read speech, which would not cover many speech variations.\n- Can you discuss the RTF? Is 0.255 or 0.200 sufficiently light?\n", "limitations": "The paper describes the limitation, especially for real-world applications. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Real-time speaker de-identification is a critical topic for voice privacy, and the motivation part of the paper clearly describes this importance.\n- The paper carefully describes the reproducibility of the proposed method.\n- Time-varying filter-based de-identification is novel compared with existing studies.\n- The experimental results clearly demonstrate the improvement from the existing studies regarding both subjective and objective evaluation metrics.\n\nWeaknesses\n- The topic is specific to speech processing rather than generic machine learning and may not gain attention from general NeurIPS audiences. I think the paper would have a better fit for the speech conferences (e.g., Interspeech and ICASSP).\n- The experimental evaluation should focus more on the latency. It would be good to have the performance vs. latency trade-off.\n- Although the paper shows the overall improvement of the proposed method, the proposed method seems to be very different from the other methods, especially for the encoder part and training schemes in addition to the time-variant\" FIR filter. For example, it is not clear about the experimental justification of this neural network architecture. The proposed method tries to disentangle various speech factors (pitch, phoneme, loudness) based on speech processing models in the encoder, but this architecture has no experimental validation. I suggest the authors provide the contribution of these factors (possibly through the ablation study of each aspect) to the overall performance.\n- I have the same discussion for the loss function. There is no experimental justification for the proposed training scheme.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SMet9MEvZ0u", "reviewer": "Reviewer_LEcn", "summary": "This manuscript proposes an adversarial attack for audio-to-audio model that preserves user’s privacy against speaker recognition models. The proposed approach is based on a seq2seq neural network model, trained with multi-task losses including an adversarial loss to attack the speaker recognition model, and an auxiliary loss for finite impulse response filtering and audio reconstruction.", "questions": "N/A", "limitations": "Satisfactory", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* The proposed approach is interesting, and the technical are enough.\n* The authors conducted both objective and subjective evaluations on the audios generated from the proposed methods, regarding the speech audio quality and the performance on (seen/unseen) victim speaker recognition models. Results show that the proposed approach significantly outperform baselines.\n* The paper is well written and well organized. References are enough. \n* Sample page and code are available.\n\nWeaknesses\n* The evaluations are only conducted on the generated audios. It would be more promising to have evaluations on a couple of real audio-to-audio tasks like enhancement, voice conversion, speech-to-speech translation. Although the generated audios have satisfactory quality from the evaluations on themselves, it is possible that the extracted features are very different since the proposed approach modulates their waveforms, causing regressions on these tasks \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2o0MOSAXm0W", "reviewer": "Reviewer_G7js", "summary": "This paper presents VoiceBox, an adversarial audio attack which (1) dramatically lowers the accuracy of speaker recognition systems, (2) is mostly imperceptible to humans, and (3) can operate in real-time on live audio streams. The authors further demonstrate that their proposed model may also transfer to speaker recognition systems that it was not explicitly trained to fool.", "questions": "Can the authors could shed more light on this contradiction in the rebuttal, or otherwise rework their submission to address this concern in a more transparent fashion?\n\nOther comments / questions:\n- Especially in a paper largely concerning privacy, a section devoted to Ethics would be nice to see. While privacy is an admirable goal, might there be some potential ethical *downsides* of releasing a de-identification system like VoiceBox?\n- Seems to be a known issue already, but the author should rework their results to include the ASR results which are only present in the online supplement\n- Clarify code release? The checklist indicates that code is proprietary and will not be released, but the supplement implies that it will be. (My review gives the benefit of the doubt that the code will be released)\n", "limitations": "See above", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this is an interesting and well-written paper which proposes a nice method and demonstrates its efficacy through carefully-designed experiments and user studies. Moreover, the authors plan to release code and a reference implementation of the system which can run on commodity hardware. My main issue with this paper lies in a perceived contradiction in the core motivation (expanded below). However, despite this contradiction, I do think VoiceBox will be of interest to the voice privacy research community in both the short and long term, as well as possibly even to users in the short term.\n\n**Contradiction in core motivation.** There is seemingly a large contradiction in the premise of this work: that in a landscape of automated surveillance via speaker recognition, we should work to design attacks that are imperceptible to humans, i.e., ones that _preserve the identity_ of the speaker to human listeners. I cannot understand why this is a desirable property. Suppose that, once VoiceBox is released, speaker recognition systems are augmented with a defense which detects if speech was modified by VoiceBox. In this case, users could face repercussions such as _denial of service_ (automatable) or _targeted retaliation_ (through manual audit to determine identity). In the worst case, if speaker recognition systems can also be trained to be  _robust_ to VoiceBox, users may even face _automated_ targeted retaliation, since both their use of VoiceBox and their identity would be known automatically.\n\n**Why not voice conversion?** Given this contradiction, it seems to me that _voice conversion_ is a much more desirable goal, because it would at least remove the possibility of targeted retaliation against users (assuming that the voice conversion process eliminates the possibility of recovering original speaker identity). The authors mention voice conversion in related work, advocating in favor of their approach as it is “inconspicuous and minimally invasive to the user experience”. However it is historically unlikely that any attack which is “inconspicuous” to humans will be undetectable by a deep learning system, and it’s unclear what the “user experience” means in this context, so this justification is largely unsatisfying.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept