Synthesising Audio Adversarial Examples for Automatic Speech Recognition
Authors: Xinghua Qu
pengfei wei
Mingyong Gao
Zhu Sun
Yew-Soon Ong
Zejun MA
Venue: ICLR 2022 Submitted (2022)
Keywords: Adversarial Attack
Speech Synthesise
Automatic Speech Recognition

Abstract: Adversarial examples in automatic speech recognition (ASR) are naturally sounded by humans yet capable of fooling well trained ASR models to transcribe incorrectly. Existing audio adversarial examples are typically constructed by adding constrained perturbations on benign audio inputs. Such attacks are therefore generated with an audio dependent assumption. For the first time, we propose the Speech Synthesising based Attack (SSA), a novel threat model that constructs audio adversarial examples entirely from scratch, i.e., without depending on any existing audio) to fool cutting-edge ASR models. To this end, we introduce a conditional variational auto-encoder (CVAE) as the speech synthesiser. Meanwhile, an adaptive sign gradient descent algorithm is proposed to solve the adversarial audio synthesis task. Experiments on three datasets (i.e., Audio Mnist, Common Voice, and Librispeech) show that our method could synthesise audio adversarial examples that are naturally sounded but misleading the start-of-the-art ASR models. The project webpage containing generated audio demos is at https://sites.google.com/view/ssa-asr/home. 

Under review as a conference paper at ICLR 2022
SYNTHESISING AUDIO ADVERSARIAL EXAMPLES FOR AUTOMATIC SPEECH RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial examples in automatic speech recognition (ASR) are naturally sounded by humans yer capable of fooling well trained ASR models to tran- scribe incorrectly. Existing audio adversarial examples are typically constructed by adding constrained perturbations on benign audio inputs. Such attacks are therefore generated with an audio dependent assumption. For the first time, we propose the Speech Synthesising based Attack (SSA), novel threat model that constructs audio adversarial examples entirely from scratch, i.e., without depend- ing any existing audio) to fool cutting-edge ASR models. To this end, in- troduce a conditional variational auto-encoder (CVAE) as the speech synthesiser. Meanwhile, adaptive sign gradient descent algorithm is proposed to solve the adversarial audio synthesis task. Experiments on three datasets (i.e., Audio Mnist, Common Voice, and Librispeech) show that method could synthesise audio adversarial examples that are naturally sounded but misleading the start-of-the- art ASR models. The project webpage containing generated audio demos is https://sites. google. com/ vwt./vieii////////e ssa-sssaaasssssss asr/home
INTRODUCTION
Deep neural network (DNN) based models have documented many success stories in various do- mains, such as reinforcement learning (Silver et al., 2017). image classification (Deng et al. 22009). and automatic speech recognition (ASR) (Chan et al. (016)). However, DNN models are found be vulnerable to adversarial attacks (Goodfellow et al. 2015). viz., the slightly perturbed input would cause severe errors or performance drops of well trained DNN models. This paper mainly focuses on the ASR domain, where many voice assistant systems could be hijacked or controlled by the audio adversarial examples constructed by an attacker.
To investigate the threat of audio adversarial examples, many different approaches (Carlini & Wag- ner 2018; Yuan et al., 2018, Yang et al.) 2019 Qin et al., 2019) have been developed. In general, existing approaches assume that the semanties (for human beings) of adversarial audio can be pre- served as long as the distance between the adversarial audio and the benign audio is restricted prop- erly. In particular, shared design principle is add constrained adversarial perturbations (i.e., as imperceptible as possible) on benign audios yet with the goal of fooling ASR models significantly. For instance, QQn al. (2019) introduced psychoacoustic rule of auditory masking only add perturbation on a benign audio where the noise is hard to be heard. Therefore, existing approaches can be deemed as audio dependent attack (ADA), viz., the audio adversarial examples have to be constructed depending on some benign audios as shown in Figure (a). However, in real cases, the human speaker and/or the benign audio may not be available or accessible, Moreover, ADA relies on an imperceptible perturbation principle, viz., the added perturbation must be restricted enough to avoid being perceived by human beings.
In contrast, this paper proposes the audio independent attack (AIA) shown in Figure I (b) that sheds light on a more general principle of adversarial attacks, viz., any audio that deceives ASR models yet fails to deceive human beings would cause security issues in speech recognition. Alter- natively stated, an adversarial audio does not have to be constructed based on an existing benign audio as ADA does. Our AIA thus enables a novel threat model that constructs audio adversarial examples completely from seratch instead of adding perturbations on existing benign audios. Par- ticularly, mounted recent advances in neural speech synthesis (Tan al. 2021),
directly

---

Under review as a conference paper at ICLR 2022
"Tue"
MW Barigh audio
M Advensania audio
' ASR model
M Adversania audio
"True"
Neural Speech Synthesis
Perturbation,
(a) Audio dependent attack
"False"
(b) Audio independent attack
Figure 1: Audio dependent attack versus audio independent attack.
synthesise the adversarial audio that conserves the desired semantic content but deceives the ASR model to predict incorrect or even targeted transcriptions.
With this goal in mind, propose the Speech Synthesising based Attack (SSA) as shown in Figure 2 where conditional variational au- toencoder CVAR is incorporated to synthesise audio waveform r that is connected to ASR model thereafter. The basic philosophy of SSA is: the audio style vector z in CVAE controls the pitches and rhythms of synthesised wave- forms that may cause the ASR model to tran- scribe incorrectly or even as target; thus some particular synthesised waveforms would be- come adversarial examples. In particular, fol- lowing (Kim et al. 202T), we first train CVAE by adopting variational inference augmented with normalizing flows and an adversarial train- ing process,
"Send greating email to TOM"
"Transfer one million dollars to Jerry"
Ctext
Transcription Decoder
Text Encoder
LSSA (Ceert 2)
Duration Predictor
Projection
Audio Style Vector 2
ASR Model F(-)
Flow E
to synthesise natural sound- ing audios. Once the CVAE is well trained, given a sequence of conditional text that repre- sents the ground truth audio semantics, we can optimize to get particular that can de- ceive an advanced ASR model through reg- ularized connectionist temporal classification loss. In this regard, we formulate the adversarial example synthesising task gradient sign based optimization problem. More importantly, to solve the optimization efficiently, we design an adaptive learning rate decay based on the annealing mechanism, which automatically adjust the step size during optimization. Our contributions summarized as:
Decoder
Raw Waveform
8Cg8A (Stesst
8s
Figure 2: The general structure of SSA.
For the first time, we propose an audio independent adversarial attack as a novel threat model in ASR, which constructs adversarial example completely from scratch without depending any existing speaker/audio. This sheds light on a more general principle of adversarial attacks, viz., any audio that deceives ASR models yet fails deceive human beings would cause security issues in speech recognition.
We develop SSA that adopts CVAE as the speech synthesiser. To efficiently synthesise audio adversarial examples, we establish an adaptive sign gradient descent algorithm via designing an annealing mechanism inspired learning rate decay.
Extensive experiments across three datasets (i.e., Audio Mnist (Becker al. 2018), Common Voice (Ardila et al. 2019), and Librispeech (Panayotov et al. (0155) based on DeepSpeech model (Amodei al. (016)) show that our SSA can effectively generate audio adversarial examples.
2 RELATED WORK
In ASR adversarial attacks, Carlini & Wagner 2018 were among the first ones to showcase that slight perturbations on audio can easily fool a sf of the-art art ASR model to transcribe the perturbed audio into any target sentence. Later on, Yuan et al. (2018) demonstrated that such adversarial audio perturbation can embedded into a song. Yang et al. (2019) analyzed that the temporal dependency could promote the discriminative power against adversarial examples in ASR. Moreover, Khare al.
2

---

Under review as a conference paper at ICLR 2022
2019) and Taori et al. (2019) found that similar audio adversarial perturbations could be generated
using black- -ox optimization algorithms (e.g., genetic algorithm (Mitchell| 1998) ). To generate the audio adversarial examples faster and improve the attack efficiency, Liu et al. (2220) proposed to adaptively rectify the weights of audio perturbations on different positions. To make the audio perturbation more imperceptible, QQin et al. (2019) introduced psychoacoustic principle of auditory masking to smartly add perturbations on frames where noises are hard to be perceived. Xie et al. (202I) utilized generative adversarial network (GAN) to generate adversarial examples on speech domain, where GAN is to learn the distribution of predefined adversarial perturbations. Thus the adversarial examples are still generated depending on the benign audios.
In summary, previous adversarial attacks on ASR are audio-dependent, viz., the audio adversarial examples must be generated based on corresponding benign audios. Although there are studies, e.g. (Song et al., 2018; Wang et al. 2019). on generating adversarial examples from scratch, but most of them focus on the image domain. Carlini & Wagner] (2018) studied an audio attack that starts from non-speech (e.g., a piece of classic music), while existing audios are stilled required to mount perturbations on. Roy et al. 2018)) and Zhang et al. (017)) proposed to modulate voice commands on ultrasonic carriers (e.g., frequency 20 2H2) to achieve inaudibility, where the scenario is an adversary that stands on the road and silently controls the voice command assistant systems. This therefore is different from our scope of generating natural sounding adversarial audios. To the best our knowledge, our work is the first attempt to generate audio adversarial examples from scratch without utilizing benign audios in ASR.
BACKGROUND
Automatic Speech Recognition (ASR). In this paper, we focus on the speech-to-text tasks that are based neural ASR models. Following previous studies (Esmaeilpour et al.] 2021; Liu et al., 2020 Yakura & Sakuma 2019; [Carlini & Wagner 2018), we stick our attacks on the Deep- a (016), state-of ASR model based Connectionist Temporal Clas- sification (CTC) method (Graves et al. 2006). DeepSpeech uses CTC as the input aduios and the corresponding transcriptions are unaligned, viz., the exact position of each word in the audio sample is unknown. To enable efficient supervised training, a transcription is first enumerated to obtain all alignments. Hence, the CTC loss is minimized maximize the probabilities over all alignments.
Speech (Amodei et al.]
CTC Loss. Let X' be the audio imput domain and y be the text output domain with dimension [
The ASR model is donoted as F X + y>N-11,, which takes N frames I E X' as input and outputs probability distribution F(x) over the output domain. Given y as phrase (i.e., a sequence of characters), we define a token sequence π being reducible to y if the two operations, namely, removing sequentially duplicated tokens, and deleting all blank tokens on π could produce y. We further denote π as an alignment of if can be reduced to y and the length of equals to the length of prediction F(x)). Let +Dy) be the alignment set obtained from targeted transcription y using dynamic programming (Graves et al. 2006). Accordingly, the CTC loss can be formulated as,
L -lgg
II. F
where F(x); is the probability of the token ग. on the ith frame. To get the transcription in inference, a decoder DD (e.g., greedy decoding or beam search decoding) is required. Thereby, if an ASR model is well trained, the transcription would satisfy y = D(F(x)) = f(x), where f(-) is merged denotation of F and D.
White-Box Assumption. Following most of previous studies, we use the similar white-box assump- tion, viz., the parameter and structure of ASR model known. Investigating the black-box attack is another direction. Morcover, there many ways (Oh et al.) 2019 Canella-Beguell et al. 2021 Qin et al. 2019) to convert a black-box model to a white-box one.
Targeted Attack. Compared with the untargeted attack that only maximizes the word error rate (WER), the targeted attack is more challenging task since it requires not only the audio perturbation imperceptible, but also the adversarial audio being transcribed to a specified target phrase. Our main focus is on the targeted scenario. Without specification, the adversarial example in following sections refers to the targeted one.
3

---

Under review as a conference paper at ICLR 2022
Speech Synthesise (End-To-End Text-To-Speech). Text-to-speech (TTS) model synthesises wave- forms given text phrases as semantic contents. For efficiently synthesising adversarial audios in our SSA, we turn to the end-to-end TTS model that could easily utilize modern parallel processors for faster synthesis speed. In particular, we utilize a conditional variational autoencoder (Kim et al, 202T) based TTS model generate natural sounding audios, which is explained in Section 4.2
METHODS
4.1
PROBLEM SETTINGS
Given a well trained ASR model f(.), the objective of adversarial attack is to construct an audio waveform x X that is naturally sounded yet able deceive f(-) in predicting incorrect/targeted transcriptions. Suppose o X' + y is an oracle that takes an audio waveform x as input and outputs the ground truth transcription yo = o(x), where y is the set of all text transcriptions under consideration. Moreover, compared to the untargeted attack that only introduces spelling errors, we focus on the more challenging targeted attack, viz., f(z) y where y E y is the expected target transcription by an attacker.
In previous studies, the adversarial audio x constructed by adding perturbation on benign audio To. viz., x + å; thus being dependent on T. Mounted on these notations, we give a formal definition of the previous audio dependent attack as follows.
Definition (Audio Dependent Attack ADA). Given benign audio F, and its oracle transcription
No = ol (. the corresponding adversarial example can be defined as any audio x, viz., E Ass - {2o 5 E X138, .(() S <No(xo + 6) = yo = o(x,) 0 nf + 6) = y f(x) # ye N + yr
where MM-) is distance measurement (e.g.. matrix norm); indicates the targeted transcription of an attacker.
small positive constant; and yr
From Definition [ the adversarial audio x is directly built on benign audio di.- Moreover, to guarantee x being acoustically realistic or natural, the efforts mainly focus on forcing ( S with the goal of restricting æ to be close to To. However, in some cases, the benign audio x, may not be available. For instance, attacker want to deceive voice commander when no human speaking happens nearby. Moreover, ADA relies on an imperceptible perturbation principle, viz., the added perturbation 5 must be small enough avoid being percepted by human beings. In contrast, more general scenario would be audio independent attack that is defined as follows.
Definition (Audio Independent Attack AIA). Given a conditional text (i.e., yo). an audio independent adversarial example can be any element from Ao for X o(x) = yo =
=y N. yr where o(x) = yo indicates that the synthesised audio æ cor-
rectly conveys its semantic content Yoi and f(x) y/: means the ASR model f(() successfully fooled to output the targeted transcription yt-
Such AIA sheds light on a more general principle of adversarial attacks, viz., any audio that deceives ASR models vel fails to deceive humans would cause security issues in speech recognition. In other words, the adversarial audios are not necessarily constructed via adding perturbations. Instead, they can be directly synthesized with the goal of preserving the desired semantic content (i.e. o(z) yo). while simultancously deceiving the ASR model to predict incorrect or even targeted transcriptions (i.e. f(x) = y+). This motivates us leverage the powerful generative model in TTS area to construct such adversarial audio z. Next, we will introduce the speech synthesising based attack.
4.2 SPEECH SYNTHESISING BASED ATTACK
The overall structure of the speech synthesis based attack (SSA) is depicted in Figure 2
Conditional Variational Autoencoder based Speech Synthesis. The key for synthesising natural sounding adversarial audio is to model the TTS mapping. In practice, we can select different types of TTS models (Tan al. 2021 to generate natural sounding audios. We choose the recent conditional variational autoencoder (CVAE) based TTS model (Kim et al., (02TI) due to its rich variety in audio generation, viz., text input can be spoken in multiple ways with different pitches and rhythms.
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: SSA Algorithm
Input: y/ the ground truth lext; yt - the target for attack; 9 - the generative model for speech
synthesise; F(.) - the ASR model: cxo the initial learning rate; da - the learning rate decay ratio; A-the the weight of Erea(z); 1 Im - the maximum iteration number; Dza Levenshtein distance.
Initialization: audio style vector NO0, 1); the patience p O; record best loss C+ +00; Im do
for
ni E
Calculate the SSA loss Lasa(z, No: y/) in Eq. 4
if Cesa (2, No: 41) <= L* then
L+ = Lusa (2, yo: Vt)
else
L P - p+1
if Pm then L da a,
Do back-propo to get ac (2.0o.0))))) and update following Eq. 5
10
11
if (f yo)), v:) == 0 then
Return the successful audio adversarial example = G(z yo)
13 Return the current best audio adversarial example = G(2, yo)
This accordingly enlarges the sample space, and enhances the possibility of constructing successful adversarial examples. In specific, such CVAE based TTS model (Kim et al. (021) is based on three components: 1) conditional VAE formulation; 2) alignment estimation derived from variational inference; 3) an adversarial training for improving the synthesis quality. The architecture of the CVAE model in inference is shown in Figure where the corresponding training loss and settings can refer to (Kim et al. 202T).
In particular, given conditional text Gtent Yo (where Yo conveys the ground truth semantic content), the TTS model aims to build the mapping G(.), viz.,
x G(z, Ctezt), 2 (),
where is normally distributed vector that controls the audio styles with different pitches and rhythms. Therefore, with different 2, there are different ways to speak the content Oteat- This aligns to the fact, viz., even human beings always pronounce same words differently from time to time. Note that the audio x generated by CVAE model G(-) in (im et al.] [021] has been tested to be natural sounding using the mean opinion score obtained from Amazon Mechanical Turk (www. mt urk com). The following goal thus becomes figuring out a particular 2 that can fool an ASR model F()).
Speech Synthesising based Attack (SSA) Formulation. We focus on the targeted attack, viz., deceiving the ASR model F(+) to predict target phrase y/- Given the speech synthesiser G(2, Ctezt) in Eq. (2). our SSA is formulated as finding a particular 2* during synthesising audio x that can enable the targeted attack. To this end, a loss function Lesa is designed construct natural sounding yet fooling enough æ. Accordingly, the optimization of z can be defined as
arg arg min Lasa (z, No. ye) arg min (9t No), ye) + (2))
(3)
where is the phrase that the CVAE model wants to synthesise; is the targeted phrase that the ASR model is fooled to predict; and (z) is the regularization loss controlled by A. To be specific, Creg(2) (z) is designed to boost the generated audio to be naturally sounded. Based on (Kim al. 2021 is sampled from a normal distribution. To preserve this property, we design it as:
Grog( (z) = ((((=))) + ((((z) -1),
where ob.) is an absolute value function; fu()) and 8(2) are the mean and variance of 2, respectively. To detemine whether the generated audio ((2, [2, Mo)
enable successful targeted attack (i.e., f(x) = Yt), we involve the Levenshtein distance YYujian & Bo] 2007). During the optimization of Lusa (z, You ve), if Dis (fG((, ()))))()))))))))))))))))())()))) 36) O, the targeted attack is successful.
Adaptive Sign Gradient Descent. To solve the optimization problem in Eq. (3) we propose an adaptive sign gradient descent algorithm whereby an adaptive learning rate decay mechanism is
5

---

Under review as a conference paper at ICLR 2022
designed in the sign gradient descent optimization. Typically, the sign gradient descent is frequently adopted in adversarial attack algorithms, e.g., fast gradient sign method (Goodfellow et al., 2015)
and projected gradient descent (Madry et al. 2018). Moreover, [Balles & Hennig (2018) theoretically proved the benefits of using gradient sign as the optimization direction. In our SSA, the sign gradient descent based optimization is given by,
2 2 a OLssa (z, yo: Vs
(5)
Oz
where a is the learning rate, which significantly impacts the convergence of the SSA optimization. Inspired by the heuristics of annealing (Bertsimas & Tsitsiklis 1993), we carefully design an adap- tive learning rate decay mechanism as below.
Definition 3 (Anncaling based adaptive learning rate decay). For every Pm steps where the attack loss Lusa 22, Yo: yt) has no improvement, the learning rate a will decay as a == da a, where da E [0, 1] is the decay ratio.
Definition 3 tells that if a specific a gets stuck for particular Pm steps with regard to Lusa (.), a should decay perform more local search. The overall process of SSA is shown in Algorithm T
5 EXPERIMENTS AND RESULTS
5.1 EXPERIMENT SETTINGS
Datasets. In our experiment?' TT we use three datasets, i.c., Audio Mnist BBekker et al. 2018), Com- mon Voice (Ardila al. 2019), and Librispeech (Panayotov et al. 2015). However, different from previous studies using both the waveform and text label, we only utilize the text information due to the audio independent property of our SSA. In particular, the Audio Mnist contains the text digits (i.e., from "ZERO" to "NINE"). When building the targeted attack pairs, we choose one text digit "eg.""ZROO") Yo and enumerate the rest digits (i.e., from "ONE" to "NINE") its attack target yt- For Common Voice and Librispeech, we randomly sample 1000 text labels first. Those sampled texts are filtered and clustered based on their text length. Thereafter. 100 texts are sampled out from the filtered 1000 samples as the candidates of conditional text yo. The corresponding target text ye is sampled from their matching clusters. The length comparison between yo and y/ on the two datasets are shown in Figures [TO I1 in the appendix, where we notice that Yo and y/ are aligned with a similar length. More analysis can refer to appendix 7.1 Our constructed dataset are released to benefit future research on speech synthesise/generatiie model related attacks.
Model Settings. In our SSA paradigm, the CVAE model and DeepSpeech are utilized as speech synthesiser and speech recogniser, respectively. The CVAE model follows the setting of VITS (Kim) cct al. 2021) during inference. In particular, we utilize the CVAE model trained on VCTK dataset Veaux et al. 2017), which can be obtained from VITS The standard deviation of the input noise for the stochastic duration predictor is set to be O, thus making it a deterministic one. In addition, scaling factor for is applied. The DeepSpeech model follows the Pytorch implementation in the adversarial robustness toolbox (Nicolae et al. 2018).
Evaluation Metrics. Two evaluation metrics, i.e., the word error rate (WER) and success rate (SR), are adopted under the targeted attack setting. In particular, they are defined as,
WER S D+ SR Nw
N. No
(6)
where S, D and I indicate the number of subsitutions, deletions and insertions of words respectively; No is the total number of words; N₂ is the number of successful adversarial example (i.c., f(x) yr): and N, is the total number of audios synthesised. Note that SR is same as the sentence-level accuracy that is used in previous studies QQnn et al. 2019). In general, larger values of WER and SR
indicate a stronger attack algorithm.
SSA Optimization Settings. The learning rate decays in updating as highlighted in Definition In particular, og (i.e., initialization of learning rate) is searched in the range of
3 and Algorithm|] I
Our code will be released upon acceptance.
attps: /ttre//dddeeeeeeeeeeeeeeeeeeeeea
wtw.e/iiieeeoooeeeeeeeeooooo google. com/ ht////1l////////// htt..//oo..////eeeeeeeeeeeeeeeee
httpa:/// tttpp/////tt...ccoooo thub.comm jaywa 310//tt

---

Under review as a conference paper at ICLR 2022
Table I: Targeted results of SSA on the three datasets and comparisons with baselines.
Attack algorithms
WER (%)
SR (%)
DeepSpeech (No Attack)
7.55
NA
C&W (Carlini & Wagner 2018
78.94 2.01
30.74 1 3.16
Baselines
Y&S Yakura Sakuma
2019)
80.28 1 3.14
35.49 0.28
GAA laon ct al. 2019)
65.80 2.55
48.35 3.38
MOOA (Khare ct al. 2019)
68.06 + 2.71
47.01 + 1.42
Metamorph (Chen et al.
72.48 1.06
45.84 + 4.71
CIPMA (Esmaailpour et 2021
88.19 3.15
21.69 3.09
SSA-Audio Mnist
100.00 + 0.00
100.00 + 0.00
SSA
SSA-Common Voice
106.27 27 = 18.03
96.00 = 8.00
SSA-Librispeech
100.71 13.21
83.19 16.33
SSA-Average (Common Voice & Librispeech)
103.49 + 16.05
89.60 + 14.37
Table The MOS comparison between original and SSA synthesised audios.
Type of audios
MOS
Before attack (original synthesised audios)
4.09 - 0.10 3.39 = 0.29
After attack (SSA synthesised audios)
[0.01, 0.09] stepped by 0.01; the patience pm: ranges from 50 to 400 stepped by 50; the learning rate decay ratio de searched in {0, 5, 0.6, 0.7}; moreover, the regularization weight searched in {0, 50, 100, 150, 200}. The maximum iteration number Im is 8000, where any case without finding a successful attack within I,, budget deemed failed.
5.2 TARGETED ATTACK PERFORMANCE
We first show that our proposed SSA capable of efficiently constructing audio adversarial exam- ples, viz., significantly outperforming existing baselines in Table In particular, the baselines in- clude DeepSpeect_ [ C&W attack (Carlini & Wagner 2018). Y&S attack (Yakura & Sakuma 2019) GAA attack (Taori al., 2019), MOOA attach (Khare al. 2019), Metamorph attack (Chen al., 2020). and CIPMA attack (Esmaeilpour et al., 2021). The results from the baselines are reported in (Esmaeillour et al. 2021), which are averaged results over Common Voice and Librispeech. Our SSA is evaluated on three datasets (i.e., Audio Mnist, Common Voice, and Librispeech), where an averaged result over Common Voice and Librispeech calculated for fair comparison. All the results of our SSAs are based on targeted attack settings.
In general, we can see that our SSA achieves remarkably better performance on both WER and SR. Specifically, (1) SSA-Average achieves 103.49% and 89.60% regarding WER and SR, respectively, which outperforms the best results obtained by CIPMA on WER (i.e. 68.19%) and GAA on SR (i.e. 48.35%); (2) both SSA-Common Voice and SSA- SSA-Librispeech dramatically outperform the best results obtained from the baselines as well. Especially for SSA-Common Voice, both of its WER (i.e. 106.27%) and SR (i.e. 96.00%) are the best across all comparisons; (3) SSA-Librispeech behaves worse than SSA-Common Voice with regards to WER and SR, which is mainly attributed to the difference of conditional text lengths between the two datasets; and (4) our proposed SSA realizes 100% w.r.t. both WER and SR on Audio Mnist, which can serve as a strong baseline for future study on the audio adversarial attack. We suggest the reader listen our synthesised adversarial audios that are available at our 3
5.3 QUALITY EVALUATION SYNTHESISED AUDIOS
Although the regularization loss Lreg(2) in our SSA is designed to force our synthesised audios to be independently and identically distributed w.r.t. the audio generated in VITS [1], we still find some synthesised audios not natural sounding enough. Therefore, we further evaluate the quality of synthesised adversarial audios by mean opinion score (MOS) tests. Specifically, we invited 20 participants to rate on 50 sample pairs, where each sample pair includes original synthesised audio by CVAE and its corresponding synthesised adversarial audio optimized by our SSA. Each participant will listen to these 100 audio samples that are randomly shuffled. The naturalness rating
benchmerk4mozi lls- Ltw.eeooiiil--deeppeeeeeeel
htspa://sitess goog wttp://....oo//e//eeeeeeooo/e http://ss....ooooooooooooo///aas

---

Under review as a conference paper at ICLR 2022
50
Pow 203
100
New 992
CE
9G
53
Succes rote: (50
80
60
0.03
0.02
06 0.03
-- 04
du 0.05
o- 0.06
0.00.7 de 6.00.
6- 0.09
Fatlnce
Learing rate
Figure 3: The results of SR with different cig and Pm on Audio Mnist.
50
263
250
300
352
100
Succos rate
60-
40
20
0.01
0.00
0.03
--00.0
0.05
0.00
00007 -000 0.05
Learing rate intializaton
Figure 4: The results of SR with different ai and pm on Common Voice.
score is scaled from 1 to 5. The MOS results are shown in Table 2 where we can observe a slightly worse MOS score of our SSA synthesised audios (i.e., 3.39 0.29) compared with that of original synthesised ones (i.e., 4.09 + 0.10). This again indicates that there exists distortions in the synthe- sised audios by SSA. However, we further note that the difference of MOS scores between the two types of synthesised audios is not significant, which indicates that the distortions are still subjec- tively acceptable. Future works can focus on how to eliminate such distortions in the adversarial audio generation, such redesigning the regularization loss Creg(z) in Eq (3).
5.4 HYPERPARAMETE ANALYSIS ON THE ADAPTIVE SIGN GRADIENT DECENT
In the adaptive sign gradient decent. the initial learning rate dep and the patience Pm are two important hyperparameters. Their impacts regarding SR on Audio Mnist and Common Voice datasets are displayed in Figure 3 and respectively. Moreover, we also analyze their impacts on Common Voice w.r.t. WER. Several interesting findings are noted below.
Analysis on SR. (1) Generally, the SR first climbs up as ag increases and then keeps stable with further increase of ag. For instance, a stable SR is achieve with 0.03 on Audio Mnist. Similar trends are also held with pm. (2) On both Audio Mnist and Common Voice, Pm needs to be set as a mild value (e.g., Pm E {300, 350)) order to obtain a promising SR. This indicates that the learning rate decays too fast, the sign gradient descent optimization in [ggorithmmmmay get stuck. (3) Compared the results in Figures 43.) the SR from Common Voice is usually smaller than that from Audio Mnist
with the same settings on Og and Pm- This is mainly because the audio synthesised in Common Voice is clearly longer than that from Audio Mnist.
Analysis on WER. Fig- ure (5 shows the WER with different settings of og and
5.01
4g- 0.04
0
5.02
4.- 0.00
0.20
5.3
4. 0.00
4. 0.20
Pm- As a whole, the WER
the risteee
106.0
19.6
first goes up with the in-
105.5
creasing of both parame-
144
10.
105.0
ters, while becomes less sensitive as they continue to increase. For exam- ple, comparing across dif- ferent lines (i.e., with dif- ferent ap), we can only ob- serve a slight change on WER with ap 2 0.04. This leads to a similar conclusion with the analysis on SR, viz., proper settings of oo and Pm can easily synthesise harmful adversarial audios via SSA.
10.0
104.5
104.0
12
20.21
12
12:
1035
20.33
13
12:.3
10.01
103.0
1025
399
353
260
395
420
50
100
150
200
250
300
350
400
Patlence
Figure 5: WER with different og and Pm on Common Voice.
8

---

Under review as a conference paper at ICLR 2022
case-1
fase-66
60
cave:1
case-6
500
case-29
59
5a58-75 75
(as2 80
distance
50
1352
c350 80
400
40
300
30
20
10
1000 2000 4000 5000 6000 7000 8000
1000 2000 3000 4000
5000 6000 7000 8000 Iteration steps
Iteration steps
(a) CTC loss convergence
(b) Levenshtein distance convergence
Conditional text
Target text
case-1
case-59
case-75
case-19
case-66
case-80
70-
0.05
Text Lengit
50
Learning rete
0.04
40-
30-
0.03
20
0.02
10
0.01
39
3.9
66
75
1000
2000 3000 4000 5000 7000 9000 horattio stops
(c) Text length comparison
(d) Learning rate dynamics
Figure 6: Convergence analyses on selected representative cases on Common Voice.
Poturiation
1.00
Wordtemssteggg
0.2
0.2
0.75
dowetemeennggh
0.50
0.00
-0.2
-22
-0.5
-0.0
12003
20505
30500
40503
socon
10000
20303
30200
4030a
50000
10200
20000
30000
10500
Frama number
From number
Trame number
(a) Original andio Carlini & Wagner 2018
(b) Attacked audio
C Carlini
Wagner 2018
(c) SSA generated audio
Figure 7: The comparison on waveforms between the audio dependent attack and our SSA.
5.5 CONVERGENCE ANALYSIS
One natural question is, given a particular setting on on and Pmu how does the process of adaptive sign gradient descent look like? We choose the best setting according to the results in Figure 4 viz.. 0x- 0.06 and pm = 400, and select six representative evaluated cases to show the convergence of the CTC loss, Levenshtein distance Common Voice in Figures ba-,), respectively. In addition, to assist the analysis, the correspond text length comparison and learning rate decays are provided in Figures c-d). Due to space limitation, the overall convergence process across 100 samples related to CTC loss and Levenshtein distance and learning rate decays are respectively shown in Figures 12 14 in the appendix. Some interesting observations are noted follows.
First, from Figure and Figure 12 we can see that the CTC loss of most cases quickly converges to small value close to O. The convergence speed is especially fast at the beginning of the opti- mization. Such a phenomenon is also observed in other studies (Amodei et al. 2016) related to CTC loss based training. Second, in Figure bbb), the Levenshtein distance on most cases except case converges to O. indicating a successful targeted attack. Third, on some cases, e.g., case in Fig- Gre although the convergence curve does not stop before reaching the maximum iteration step 8000, it can be deemed as an approximately successful attack as reflected by the small CTC loss in Figure 6] a) and Levenshtein distance in Figure [((b). Fourth, the additional text length comparison in Figure 6(c) shows that the text length generally reflects the difficulty of generating successful attack, viz., a longer text usually requires more iterations in both CTC loss and Levenshtein distance convergences. Lastly, the learning rate dynamics in Figure d) also showcase that for a harder prob- lem (i.c., with a larger text length), the learning rate decays more times to exploit a solution close to successful attack. More results and analyses refer to Appendix 7.2
9

---

Under review as a conference paper at ICLR 2022
5.6 WAVEFORM PATTERN ANALYSIS
To showcase that our SSA is a more general attack compared with audio dependent attacks, we fur- ther analyse the waveform pattern of both attacks shown in Figure 7 In particular, Figure 7(a) and (b) respectively depict the original audio and the corresponding adversarially perturbed audio, where we can easily observe that the attacked audio needs to be restricted to only add minor pertur- bations. In contrast, the adversarial audio constructed by our SSA as shown in Figure[]cc) is free of such restriction, viz., the waveform can be significantly different. In sum, our SSA can be deemed as audio independent attack, which brings in more threat to ASR models in the wild. Further analyses towards the audio style vector & before and after attack of SSA are shown in Appendix 7.3 In addition, also evaluate the tranferability of the adversarial audios w.r.t another ASR model (i.e., ESPnet (Watanabe et al. 2018)) as shown in Appendix 7.4
6 CONCLUSION
This paper investigates the audio adversarial attack for ASR models. Existing attack algorithms based on an audio dependent assumption, viz., adding constrained perturbations on benign audio inputs. In contrast, we propose SSA, novel threat model that constructs audio adversarial exam- ples entirely from scratch, viz, without depending on any existing audio to fool cutting-edg' ASR models. To this end, we propose to use conditional variational auto-encoder (CVAE) as the speech synthesiser. Accordingly, the adversarial audio synthesising task is formulated as an optimization problem via searching in the hidden space of CVAE. Meanwhile, an adaptive sign gradient descent algorithm is further devised to solve the SSA optimization problem. Experiments on three datasets show that our proposed SSA can synthesise audios that are naturally sounded but deceive start-of- the-art ASR models. In our experiments, we also find that some synthesised adversarial audios do not sound as natural as those without any manipulation on 2, which thus needs future efforts to enhance the quality of adversarial audio synthesis.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End- to-end speech recognition in english and mandarin. In Proceedings of the 33th International Conference on Machine Learning (ICML), pp. 173-182. PMLR, 2016.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weher. Common voice: A massively- muuliiilyy speech corpus. arXiv preprint
2019.
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp- 404 413. PMLR, 2018.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. arXiv preprint
034488 2018.
Dimitris Bertsimas and John Tsitsiklis. Simulated annealing. Statistical Science, 8(1):10-15, 1993.
Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to- text. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 1-7. IEEE, 2018.
William Chan, Navdeep Jaitly, Quoe Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4960-4964. IEEE, 2016.
Tao Chen, Longfei Shangguan, Zhenjiang Li, and Kyle Jamieson. Metamorph: Injecting inaudi- ble commands into over-the-air voice controlled systems. In Nenwork and Distributed Systems Security (NDSS) Symposium, 2020.
10

---

Under review as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255. Icce, 2009.
Mohammad Esmaeilpour, Patrick Cardinal, and Alessandro Lameiras Koerich. Towards robust speech-to-text adversarial attack. arXiv preprini urXiv:2103.08000 2021.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2015.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning (ICML), pp. 369-376, 2006.
Shreya Khare, Rahul Aralikatte, and Senthil Mani. Adversarial black-box attacks on automatic speech recognition systems using multi-objective evolutionary optimization. Conference of the International Speech Communication Association (INTERSPEECH)) 2019.
Jachyeon Kim, Jungil Kong. and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-seeech. Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Xiaolei Liu, Kun Wan, Yufei Ding. Xiaosong Zhang, and Qingxin Zhu. Weighted-sampling audio adversarial example attack. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), volume 34, pp. 4908-4915. 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018
Melanie Mitchell. An introduction to genetic algorithms. MIT Press, 1998.
Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wis- tuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v1.2.0. CoRR, 1807.01069, 2018. URL
https:
01069
Seong Joon Oh, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 121- 144. Springer, 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Yao Qin, Nicholas Carlini, Garrison Cottrell, Ian Goodfellow, and Colin Raffel. Imperceptible, bust, and targeted adversarial examples for automatic speech recognition. In International Con- ference on Machine Learning (ICML), pp. 5231-5240. PMLR, 2019.
Nirupam Roy, Sheng Shen, Haitham Hassanich, and Romit Roy Choudhury. Inaudible voice com- mands: The long-range attack and defense. In /5th {USENIX) Symposium on Networked Systems Design and Implementation ( ([NSDI) 18), pp- 547-560, 2018.
David Silver, Julian Schritwieserr, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):34--55, 2017.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial ex- amples with generative models. Advances in Neural Information Processing Systems (NeurIPS), 1188312-333,, 2018.
Xu Tan, Tao Qin. Frank Soong, and Tie- Van Liu. A survey on neural speech synthesis. arXiv e-prints, pp. arXiv-2106, 2021.

---

Under review as a conference paper at ICLR 2022
Rohan Taori, Amog Kamsetty, Brenton Chu, and Nikita Vemuri. Targeted adversarial examples for black box audio systems. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 13-20. IEEE, 2019.
Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. Cstr vetk corpus: English multi- speaker corpus for cstr voice cloning toolkit. 2017.
Xiaosen Wang, Kun He, Chuanbiao Song, Liwei Wang, and John E Hopcroft. At-gan: An adversar- ial generator model for non-constrained adversarial examples. arXiv preprint (9XXv::904.0779; 2019.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Ren- duchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Pro- ceedings of Interspeech, pp. 2207-2211, 2018. doi: 10.21447/mmerspecch.001 2018-1456..... URL
httpp:/dxx: doi wwg/1o..//.oo. 2122211432222 2tttetteeeeettteteeeeeec... 2018 17:01881455 1456
Yi Xie, Zhuohang Li, Cong Shi, Jian Liu, Yingying Chen, and Bo Yuan. Enabling fast and universal audio adversarial attack using generative model. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pp. 14129-14137, 2021.
Hiromu Yakura and Jun Sakuma. Robust audio adversarial example for a physical attack. Proceed- ings the Twenty-Eighth International Joint Conference Artificial Intelligence (IJCAI), pp- 5341, 2019.
Zhuolin Yang, Pin Yu Chen, Bo Li, and Dawn Song. Characterizing audio adversarial examples using temporal dependency, In 7th International Conference Learning Representations (ICLR), 2019.
Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, and Carl Gunter. Commandersong: A systematic approach for practical adversarial voice recognition. In 27th {USENIX) Security Symposiun (UUSENIX) Security 18), pp. 49-64, 2018.
Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 1966):0 2007.
Santiago Zanella- Beguuli, Shruti Tople, Andrew Paverd, and Boris Köpf. Grey-box extraction of natural language models. In International Conference Machine Learning (ICML.), pp. 12278- 12286. PMLR, 2021.
Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang, Taimin Zhang, and Wenyuan Xu. Dolphi- nattack: Inaudible voice commands. In Proceedings of the 2017 ACM SIGSAC Conference Computer and Communications Security, pp. 103-117, 2017.
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: A survey, ACM Transactions on Intelligent Systems and Technology (TIST), 1 11(3):1 2020.
12

---

Under review as a conference paper at ICLR 2022
7 APPENDIX
7.1 DATASET ANALYSIS
Figure 9 shows the length distribution of 1000 samples on Common Voice and Librispeech. Note that, to better sample the adversarial attack target, we filter out data points that have too small counts. From the selected 1000 samples, we generate our targeted attack dataset, viz., matching larget that has similar length with the conditional text. Figures (IO II) illustrate the comparison of the text length between conditional text Yo and target text yt on Common Voice and Librispeech, respectively, In general, the length of No and are similar. Moreover, in both datasets, the text length has a huge variance, e.g., case versus 30 in Librispeech. Compared Common Voice to Librispeech, the length of many cases on Librispeech is much larger than the maximum length (i.e., 100) on Common Voice.
7.2 ADDITIONAL ANALYSIS ON THE CONVERGENCE OF CTC LOSS AND LEVENSHTEIN DISTANCE AND LEARNING RATE DECAY
The CTC loss convergence processes across the 100 samples on Common Voice are shown in Fig- ure 12 Moreover, we further analyse the convergence of Levenshtein distance (-) (Khare al. 2019 in Figure 13, in order to depict how does our proposed SSA succeeds. Note that the Lev- enshtein distance is calculated as (Ig( No)), 3r), which indicates the distance between the transcription on the current synthesised audio x and the target transcription y/. We also analyse the
dynamics of a in Figure 14
From Figure 12 we can see that the CTC loss of most cases quickly converges value close to O. The corresponding Levenshtein distance in Figure [ exxctly converges to on these cases, suggest- ing successful attacks most cases. For the cases that do not stop the optimization before reaching the maximum iteration step 8000, both the CTC loss and Levenshtein distance end with small val- ues, which can be deemed as approximately successful attacks based the related studies (Zhang) let al. (020)) on adversarial attack in natural language processing. From Figure [4] generally dif- ferent cases have their own learning rate schedule. On most cases, the learning rate needs to decay for at least 2 times. Only on some particular cases (c.g.. case 2), directly using the initialized a without decay can find a successful adversarial example. This implies that our designed adaptive sign gradient descent algorithm enables the learning rate to dynamically change according to the optimized loss.
7.3 ANALYSES ON THE AUDIO STYLE VECTOR
We further compare the original audio style vector (i.e., sampled from the normal speech synthesis) and the adversarial = (i.c.. optimized by our SSA loss) on three cases as shown in Figure 8 For better visualization, we only plot partial of vector z. For instance, 2 reshaped from a (192 231) matrix with dimension determined by the conditional text in Figure 2 while we only plot (2 x 231) of them as shown in Figure (a). In general, Figure shows that the original 2 and adversarial are significantly different. Namely, although both original and adversarial fluctuate around the mean 0 and share a comparable variance, their specific values for each dimension are quite different. This indicates that our SSA has more flexibility of searching for successful attack. In contrast, the optimization space of previous audio dependent attacks is restricted around the original audio waveform by a norm bound as shown in Figure 7 (b).
7.4 ANALYSES OF ATTACK TRANSFER
Our SSA is designed to be ASR model dependent. In specific, the adversarial audios are synthesized based on Deep Speech. Therefore, as expected, these adversarial examples should pose limited threat to other ASR models. To validate such a hypothesis, we mount the successfully synthesised audio attacks on ESPnet (Watanabe et al., 2018) (i.e., an attention- based encoder-decoder network). In doing so, randomly sample 30 successfully synthesised attacks (i.e., based Deep Speech), input them to the ESPnet and calculate the levenstein distance (LD) with respect to the target text, where L.D 0 indicates successful targeted attack. Results show that the success rate and L.D of
13

---

Under review as a conference paper at ICLR 2022
Table 3: The human speech recognition evaluations on the original and SSA synthesised audios.
Type of audios
Human Translation WER
Before attack (original synthesised audios)
18.52 7.46%
After attack (SSA synthesised audios)
22.30 E 5.05%
these transferred attacks ESPnet are 0% and 40.97 + 15.34, respectively. This suggests that the adversarial audios generated by SSA are hardly transferable to different ASR model.
7.5 HUMAN SPEECH RECOGNITION (SR) EVALUATION
The human speech recognition (SR) evaluation has been conducted. Specifically, we invite partic- ipants to listen to 50 sample pairs, where each sample pair includes original synthesised audio by CVAE and its corresponding synthesised adversarial audio optimized by our SSA. Each participant then writes down the corresponding translation text. The WER is calculated between the human translation and ground truth text. The averaged WER from the human evaluation is shown in Table [] which indicates that the human translation performance is only slightly impacted compared the original synthesised audios.
(a) Case
(b) Case
(c) Case
Figure& 8: Comparison of from the normal speech synthesis and the one optimized by our SSA loss.
14

---

Under review as a conference paper at ICLR 2022
175
150
140
120
125
Caunt
60
50
110. 25
120 300
130 48
50V
160. PO)
[70. 800
(80. 900
190, 100)
(60, n0
(00 2000
1100 120)
120 1400
1400 J00)
1100 2000
1200 2200
1200 2000
2008
1800 300
22%
Jan
Length range
Length range
(a) Length distribution Common Voice.
(b) Length distribution on Librispeech.
Figure 9: The length distribution of 1000 samples on Common Voice and Librispeech
Figure 10: The length comparison between condition yo and target text ye on Common Voice
beggh camparion betwon and tiee tergettt
Figure 11: The length comparison between condition text and target text yr on Librispeech
15

---

Under review as a conference paper at ICLR 2022
600
case-0
case-5
600
case
case-15
case-1 1
case-6
case-11
case-16
500
500
case-12
case-17
case
case 0
case-1 11
Case-10
CTC loss
loss
400
300
300
200
200
100
1000
2000 30004000
5000 6000 7000 8000
0
1000
2000
3000
4000
5000
Iteration steps
lteration steps
600
case-21
case-26
600
31
case:36
500
case-23
ca5e-28
casa-33
case-38
CTC loss
400
case-24
300
CTC
300
200
100
100
1000
2000
3000
4000
1000
2000
3000
3000
4000 5000 6000 7000 8000
Iteration steps
Iteration
case-40
case-45
600
700
case
case 56
case-42
case-47
case
500
case-43
case-48
600
case-53
case-S8
CTC 1055
400
case-44
case-49
loss
500
400
300
300
200
200
100
1000
2000 30004000 4000 5000 7000 8000
0
1000 4000
5000 7000 8000
Iteration steps
Iteration
600
700
cese:75
case: 61
case 66
case:71
case-76
500
600
cese:77
case-B 63
case-6o 68
500
case-73
case-78
CTC loss
400
300
200
200
100
1000 2000
300D 4000 5000 6000 7000 8000
0
1000
2000
3000
3000
4000 5000 6000 7000 8000
Iteration steps
Iteration
80
700
case-95
case-81
case-86
casa-91
cose-96
82
600
CTC loss
300
case-84
case-03
case-89
case-80
500
case
case: 98
case-99
400
200
200
100
100
1000
2000
3000
4000
5000
1000 2000 3000 3000 4000 5000 6000 7000 8000
Iteration steps
lteration
Figure 12: The CTC loss convergence of SSA on Common Voice.
16

---

Under review as a conference paper at ICLR 2022
case-5
case-10
Levershtein distance
case-1
case-6
Levenshtein distance
60
case-11
case-16
50-
case-7
case-3
case-8
case-13
case-18
40-
30-
30
20
10
1000
2000 3000
5000 600D Iteration steps
4000
7000 8000
1000
2000
3000
4000
5000
Iteration steps
60
case-20
cose:25
70
case-30
case:35
Levenshtein distance
Levenshtein distance
60
case:31
case:36
50-
rase-22
case-27
ca5e-32
case:37
case:33
case-38
40
Case-24
Case-29
ca58-34
ca5e-39
20
10
0
1000
2000
3000
4000
0
1000
2000 3000 4000
5000 6000 8000 Iteration steps
8000
Iteration steps
70
case-40
case-45
80
case-50
case-55
Levenshtein distance
60
case-42
case-41
case-47
case-46
puensheein disance
60
case:51
case-52
case:56
case-57
50-
case-48
case:53
case:50
Case-44
case-49
case-54
case-59
40
30
30
20
10
10
1000
2000 3000
40005000 Iteration steps
6000 7000 8000
0
1000
2000 3000
5000 6000 7000 8000 Iteration steps
4000
case-60 60
casc-65
Levenshtein distance
60
case-61
case-66
evenshtein distance
case-71
case-76
casc-67
case-72
50-
tase-63 63
case-60
50
case-73
case-78
40
case
ca5c-69
40
30-
30
20
10-
1000
2000
3000
40005000 Iteration steps
6000 7000 8000
2000
3000 4000
5000 Ittraaion steps
7000 8000
case-80
case-85
case-90
case:95
Levenshtein distance
Case-81
case-86
en distance
case-91
case-96
case-82
case-87
40:
Case-83
case-88
60
case-93
case-98
30
40
20-
20
10
1000
3000
5000
0 1000 2000 3000 4000 5000 6000 7000 8000 lteration steps
Iteration steps
Figure 13: The Levenshtein distance convergence of SSA on Common Voice.
17

---

Under review as a conference paper at ICLR 2022
0
Ga011
(236-2 6236-
caed c2c3
c19611
(600:34
ca00-15
csm-10
0.060
0.050
CTC loss
0.03
0.040
0.035
0.01
0.025
0.020
2000
3000
6000 Iteration steps
4000
6000 7000
1000
2000
3000
4000
5000
tteraion steps
Le00
1220-24
cuee
010-30 50
33
0.060
0.055
0.045
CTC loos
0.04
0.040
0.03
0.035
0.030
0.02
0.01
0.020
1000
2000
3000
4000
1000
2000
4000 5000 6000 7000 8000 Iteration steps
steps
(3644
(350-43
case-b 45
care
035443
03955 51
53
Clile 55
case
0.05
CTC loss
0.04
0.03
0.02
0.02
0.01
1000
2000 3000 4000 5000 6000 7000 8000 Iteration steps
1000
2000 3000 4000 5000 6000 7000 aD00 Iteration steps
care-54
Cw 66
089470
0346-72
2506 63
55
CM
91
0a40-3 73
Che 75
<sc.-7 77
0.06
0.06
0.05
0.05
CTc
CTC loss
0.04
0.03
0.02
0.01
1000
2000 3000 4000 5000 6000 7000 8000 Iteration steps
1000
2000 3000 5000 6000 7000 8000 Iberation steps
<ase-30
case-34 case 35
came
<as0 60
cass 92
Cluna 64
cace 26 <ase
case
(30+1
635e 83
Use
0354
0300 91
CB60 95
Case 95
<ase
0.06
0.06
0.05
0.05
CTC 035
0.03
0.03
1000
2000
3000
4000
5000
1000 2000 300D 4000 4000 5000 7000 8DO0
Iteration steps
steps
Figure 14: The learning rate decay of SSA on Common Voice.
18


Reviews:
[{"review_id": "xaNRws0DlSv", "reviewer": "Reviewer_Hbzp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- The paper does provide a new adversarial attack mode that is different from the audio \"dependent\" attack (ADA) methods. While the ADA methods are usually performed by perturbing existing audio signals to the degree that human perception doesn't notice the difference, the proposed model can synthesize new speech signals that sound totally different from what they are intended to fool the ASR models with. \n\n- After checking out some of the demo signals posted on the anonymized website, the reviewer found that the synthesized speech signals are indeed in good quality, i.e., they do sound natural and convey the original meaning, y_o.\n\n- The targeted attack results are substantially better than the other models. \n\nWeakness:\n\n- It is not too clear to me what is the real-world attack scenario based on the proposed method. For example, I get the idea of ADA where an attacker secretly adds inaudible perturbation noise to the user's speech, so that the ASR system fails to recognize the voice command, e.g., for voice assistant services. However, what is the point of synthesizing new utterances from the attacker's point of view? The paper does not discuss this point of view, although it must nicely justify why the synthesis-based attacks are useful. \n\n- There have been other adversarial models that are based on synthesized audio. For example, \"hidden voice commands\" [a], \"inaudible voice commands\" [b], and \"DolphinAttack\" [c] are the systems that are based on synthesized attack signals. While the latter two are to attack the system via inaudible sound (i.e., in the ultrasound frequency) and the hidden voice commands are basically noise signal which is hard to tell. I understand the difference between the proposed model and these existing models. But, the proposed model is lacking a discussion as to how the attacker can successfully use the synthesized voice, while the other afore-mentioned models are with convincing and clear attack scenarios. The paper might benefit from additional justification. \n[a] https://people.eecs.berkeley.edu/~daw/papers/audio-dls18.pdf\n[b] https://www.usenix.org/conference/nsdi18/presentation/roy\n[c] https://dl.acm.org/doi/10.1145/3133956.3134052\n\n- The authors claim that the generated attack signal sounds natural and can be perceived as the ground-truth label y_o. While the reviewer also observed that the samples uploaded on the website are convincing, I believe that a large-scale evaluation on (a) sound quality of the synthesized speech (b) human annotator's transcription is necessary. The authors indeed pointed out in the conclusion section that there are unnatural examples they observed. \n\n- Related to the previous point, I believe that the loss function eq (3) isn't well justified. The first term is only to make sure that the produced speech signal is recognized as y_t, which is for the targeted attack part. However, the regularization part is just to make sure that the latent variables are following a standard normal distribution. Although I can see that the model is based on a CVAE voice synthesizer, and such regularization can indirectly enforce the model to still behave like a speech synthesizer as pre-trained. However, it is still not too clear if it is the best way. \n\n- I am curious to know how the proposed model performs in the noisy/reverberant environment. Can it still fool the ASR system if the synthesized speech itself is contaminated by real-world acoustic perturbations?\n\n- The proposed method is based on the white-box attack. Additional discussion on the black-box attack cases my strengthen the paper. \n\nOther minor comments\n\n- Definition 1 needs more elaboration. Hard to follow what the set is defined by due to the many intersection operations. \n", "summary_of_the_paper": "The paper presents a new adversarial attack mode to ASR systems based on speech synthesis. The main idea is to synthesize speech utterances that naturally sound like ground-truth transcription y_o, but can fool an ASR system. It is a targeted attack model, where each attack signal has its own target label y_t. The paper also presents a CVAE-based speech synthesis system and a learning rate decay method for training. ", "summary_of_the_review": "I found this paper novel (to my best understanding) and interesting in that it provides a new adversarial attack mode. I wish that the paper is better justified as to when this kind of attack can be a real threat, instead of relying on the reader's own conjecture. The paper is missing discussions on the comparison to the other AIA methods. The demo signals sound natural, but it might be better to include some subjective test results as the optimization on the \"naturality\" part isn't too clear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "v9xD6_byWyV", "reviewer": "Reviewer_Krqb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, clear, the problem stated concisely, ample references\nare provided, and the experiments are convincing. This novel type of attack -\nsynthesizing audio instead of perturbing - is interesting as one may not\nhave access to ground truth audio to perturb in an attack. Additionally, while\nthe overall approach is in theory complicated as it involves a STT and TTS system\nand optimized for a specific loss, the authors present their method clearly.\n\nThe quantitative presentation is good and the inclusion of their adaptive sign\ngradient decent algorithm may be useful for further researchers.\n\nThe authors show the validity of their SSA approach by demonstrating its superior\nability to attack an ASR system. Although, as the authors state, this is maybe\nnot so surprising given the fast that their SSA approach is able to modify\nthe input in many ways, whereas the perturbation approaches are limited in\nhow they can modify the input.\n\nThe only minor point is that there was no MOS done on samples the authors \ngenerated. And, as the authors point out, and as this reviewer was able to listen,\nsome of the audio samples do not sound well. But, given the large scope of\nthe work, this can maybe be done in the future.", "summary_of_the_paper": "In this work the authors explore a new vein of research in the area of audio\nadversarial attacks wherein instead of focusing on perturbing existing audio,\nthey synthesize audio. To their best knowledge, and this reviewer, this is \nthe first such attempt. To this end, the authors use a conditional\nvariational auto-encoder as a speech synthesis (TTS) model and develop \na adaptive sign gradient decent algorithm in order to solve their SSA\noptimization problem. Experiments are provided showing the effectiveness of \ntheir approach, along with a detailed analysis of results.\n", "summary_of_the_review": "Well written, novel contribution, good references, and compelling and complete experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TXYdBst8O-F", "reviewer": "Reviewer_4HAP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The most interesting component of this paper is the fact that adversarial attacks can be constructed by manipulation of the prosodic characteristics of synthetic speech.\n\nHowever, there are a number of unanswered questions about this adversarial attack.\nFirst and foremost, it is not at all clear that the use of synthesis and prosodic manipulation is a more compelling attack than adding adversarial noise to carrier speech.  The context where there is no available carrier speech is unconvincing -- an attacker can record or find some short speech transcript as easily (if not much easier) than they can train a TTS model that can be updated by CVAE.\n\nThe white-box assumption is reasonable for this kind of proof of concept, but it's less convincing than a black-box attack.\n\nThere is no assessment of TTS quality either of the un-modified initial utterance (that should be recognized correctly by the target ASR model) or the adversarial utterance.  The adversarial examples are fairly intelligible as the original phrase, but the naturalness is substantially impaired (i.e. to my ear many are obviously synthetic speech).  This is a limitation especially in comparison to audio dependent attacks (ADA) which sound like natural speech with a small amount of background noise.  For this attack to be completely successful the synthetic audio should be indistinguishable from human speech.  This isn't assessed.\n\nSmaller questions:\n* Section 1: Please elaborate on the use of Google Assistant and Apple Siri that are 'safety-critical'\n* Section 4.1: In what case is 'benign audio unavailable'\n* Section 4.1: the claim that this objective function \"guarantees [that] the audio x [is] natural sounding\" is over stated.  TTS aims for natural sounding speech, but this is far from guaranteed.\n* Section 4.2: why is dependency on some ground truth text preferable to dependency on some ground truth audio?\n* Section 4.2: what is the rationale for including levenstein distance as distinct from WER?\n* Section 5.1: there is no evaluation of Synthesized Audio Quality\n\n", "summary_of_the_paper": "This paper describes a white-box attack on ASR systems using TTS.  The main technique is (somewhat) standard adversarial training with the constraint that the CVAE component of a TTS model is the only component that is updated.  Since this component is responsible for prosodic and other non-segmental qualities of speech synthesis, the expectation is that the naturalness of the synthesis will not be impacted, but the ASR recognition will be modified.", "summary_of_the_review": "Technically this paper is sound. However, the contribution here is rather narrow.  It is interesting, but not especially consequential, that CVAE manipulation can be used to localize the contribution of an adversarial attack.  It is unclear that the presence or absence of a carrier speech signal in favor of a carrier tts utterance is a consequential modification of this attack.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S9AYLEb2q7", "reviewer": "Reviewer_aMq5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(1) The main contribution of this work is to extend the previous work by replacing audio inputs with the style vector z in the CVAE speech synthesizer. The framework is new, and the results are promising. However, the scientific depth seems a bit shallow. Since ICLR is a top machine learning conference, I would suggest the authors further highlight the major contributions, especially the scientific novelty, of this work and potential impacts to related research fields.  \n(2) Additional experiments should be conducted to confirm the effectiveness of the proposed approach. In previous works, such as (Carlini & Wagner, 2018), an audio input is given, and the goal is to modify the input so that human listeners cannot perceive the modifications while ASR performance is considerably affected. On the other hand, the SSA framework directly generates audio samples from the text. Thus, an additional experiment is required: using another ASR system, trained from other datasets or constructed by another model architectures (such as LAS or hybrid CTC/Attention), to recognize the adversarial audio samples synthesized by SSA. We expected to see that the adversarial audio samples can only attack the target ASR system while not affecting the other ASR systems.\n(3) The authors state that “Note that the audio x generated by CVAE model G(.) has been tested to be natural sounding using the mean opinion score obtained from Amazon Mechanical Turk” Please show the results of listening tests in the paper.\n(4) Since the goal of this work is to attack ASR systems, it is meaningful to show that human recognition performance is not affected. Please conduct additional experiments and include the results in the paper.\n(5) Figure 7 compares different adversarial samples. Since the samples generated by (Carlini & Wagner, 2018) and SSA are from different sources, it is reasonable that the samples show very different patterns. To us, the comparison does not provide useful information. It would be more meaningful to show and compare the vector z trained from the loss of normal speech synthesis and the SSA loss.\n(6) It is very nice that the authors provide demo samples on a website. By listening to the samples provided on the website, however, we note that the samples generated by SSA present perceivable distortions. The quality is worse than the ones of (Carlini & Wagner, 2018). I am not sure whether the distortions come from the speech synthesizer or SSA loss. Please comment on this or provide additional experimental results using another speech synthesizer.", "summary_of_the_paper": "This paper presents a speech synthesising-based attack (SSA) framework that generates audio examples entirely from scratch to attack automatic speech recognition (ASR) systems. A conditional variational auto-encoder (CVAE) based speech synthesiser is trained based on a combined connectionist temporal classification (CTC) and regularization loss. As compared to existing methods, the proposed approach serves as the first work that generates adversarial samples without audio inputs. The results confirm that the proposed SSA framework can attack the ASR system successfully. A comprehensive analysis of different hyper-parameters to the achievable performance is provided. Moreover, demo audio samples are provided by a website link. \n", "summary_of_the_review": "The promising results confirm the effectiveness of the proposed SSA to attack the ASR system. The authors also provide a comprehensive analysis of hyper-parameters setups to the achievable performance. Moreover, demo audio samples are provided by a website link. The paper is well-written, and the theoretical part should be correct. However, we think additional experiments should be conducted before the paper can be accepted for publication. Moreover, the authors are suggested to summarize the major theoretical contributions on this study and discuss potential impacts to related research fields", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a way of generating adversarial speech samples to attack an ASR system based on speech synthesis.  The proposed approach, by the name of Speech Synthesizing based Attack (SSA) , does not rely on real speech to create adversasial samples but rather create them purely from text using  a conditional variational auto-encoder.  Experiments are conducted on three datasets and the results appear to support the effectiveness of the proposed approach. 

All reviewers consider the paper technically sound and well written. Overall, the work is interesting. It pursues another possibility for a threat model on ASR and may inspire related work in the community. Reviewers also raised a number of concerns most of which have been cleared by the authors' rebuttal.  However, there are two major ones standing. One is the perceptual quality of the synthesized speech and the other is the justification of the application scenarios in the real world.  In the follow-up MOS experiments, there seems to be a noticeable difference (4.09 vs 3.39) which means the synthesized speech does sacrifice quality.  From the real-world application perspective, the scenarios proposed by the authors seem to be a bit contrived.  These two drawbacks are considered significant and need further investigation and justification.
Decision: Reject