SYNTHESISING AUDIO ADVERSARIAL EXAMPLES FOR AUTOMATIC SPEECH RECOGNITION
Year: 2021

Abstract:
Adversarial examples in automatic speech recognition (ASR) are naturally sounded by humans yet capable of fooling well trained ASR models to transcribe incorrectly. Existing audio adversarial examples are typically constructed by adding constrained perturbations on benign audio inputs. Such attacks are therefore generated with an audio dependent assumption. For the first time, we propose the Speech Synthesising based Attack (SSA), a novel threat model that constructs audio adversarial examples entirely from scratch, i.e., without depending on any existing audio) to fool cutting-edge ASR models. To this end, we introduce a conditional variational auto-encoder (CVAE) as the speech synthesiser. Meanwhile, an adaptive sign gradient descent algorithm is proposed to solve the adversarial audio synthesis task. Experiments on three datasets (i.e., Audio Mnist, Common Voice, and Librispeech) show that our method could synthesise audio adversarial examples that are naturally sounded but misleading the start-of-theart ASR models. The project webpage containing generated audio demos is at https://sites.google.com/view/ssa-asr/home.

heading: 1 INTRODUCTION
text: Deep neural network (DNN) based models have documented many success stories in various domains, such as reinforcement learning (Silver et al., 2017), image classification (Deng et al., 2009), and automatic speech recognition (ASR) (Chan et al., 2016). However, DNN models are found to be vulnerable to adversarial attacks (Goodfellow et al., 2015), viz., the slightly perturbed input would cause severe errors or performance drops of well trained DNN models. This paper mainly focuses on the ASR domain, where many voice assistant systems could be hijacked or controlled by the audio adversarial examples constructed by an attacker.
To investigate the threat of audio adversarial examples, many different approaches (Carlini & Wagner, 2018; Yuan et al., 2018; Yang et al., 2019; Qin et al., 2019) have been developed. In general, existing approaches assume that the semantics (for human beings) of adversarial audio can be preserved as long as the distance between the adversarial audio and the benign audio is restricted properly. In particular, a shared design principle is to add constrained adversarial perturbations (i.e., as imperceptible as possible) on benign audios yet with the goal of fooling ASR models significantly. For instance, Qin et al. (2019) introduced a psychoacoustic rule of auditory masking to only add perturbation on a benign audio where the noise is hard to be heard. Therefore, existing approaches can be deemed as audio dependent attack (ADA), viz., the audio adversarial examples have to be constructed depending on some benign audios as shown in Figure 1 (a). However, in real cases, the human speaker and/or the benign audio may not be available or accessible. Moreover, ADA relies on an imperceptible perturbation principle, viz., the added perturbation must be restricted enough to avoid being perceived by human beings.
In contrast, this paper proposes the audio independent attack (AIA) as shown in Figure 1 (b) that sheds light on a more general principle of adversarial attacks, viz., any audio that deceives ASR models yet fails to deceive human beings would cause security issues in speech recognition. Alternatively stated, an adversarial audio does not have to be constructed based on an existing benign audio as ADA does. Our AIA thus enables a novel threat model that constructs audio adversarial examples completely from scratch instead of adding perturbations on existing benign audios. Particularly, mounted on recent advances in neural speech synthesis (Tan et al., 2021), we can directly
synthesise the adversarial audio that conserves the desired semantic content but deceives the ASR model to predict incorrect or even targeted transcriptions.
With this goal in mind, we propose the Speech Synthesising based Attack (SSA) as shown in Figure 2, where a conditional variational autoencoder CVAE is incorporated to synthesise audio waveform x that is connected to an ASR model thereafter. The basic philosophy of SSA is: the audio style vector z in CVAE controls the pitches and rhythms of synthesised waveforms that may cause the ASR model to transcribe incorrectly or even as a target; thus some particular synthesised waveforms would become adversarial examples. In particular, following (Kim et al., 2021), we first train CVAE by adopting a variational inference augmented with normalizing flows and an adversarial training process, so as to synthesise natural sounding audios. Once the CVAE is well trained, given a sequence of conditional text that represents the ground truth audio semantics, we can optimize z to get a particular z∗ that can deceive an advanced ASR model through a reg-
ularized connectionist temporal classification loss. In this regard, we formulate the adversarial example synthesising task as a gradient sign based optimization problem. More importantly, to solve the optimization efficiently, we design an adaptive learning rate decay based on the annealing mechanism, which can automatically adjust the step size during optimization. Our contributions are summarized as:
• For the first time, we propose an audio independent adversarial attack as a novel threat model in ASR, which constructs adversarial example completely from scratch without depending on any existing speaker/audio. This sheds light on a more general principle of adversarial attacks, viz., any audio that deceives ASR models yet fails to deceive human beings would cause security issues in speech recognition.
• We develop SSA that adopts CVAE as the speech synthesiser. To efficiently synthesise audio adversarial examples, we establish an adaptive sign gradient descent algorithm via designing an annealing mechanism inspired learning rate decay.
• Extensive experiments across three datasets (i.e., Audio Mnist (Becker et al., 2018), Common Voice (Ardila et al., 2019), and Librispeech (Panayotov et al., 2015)) based on DeepSpeech model (Amodei et al., 2016) show that our SSA can effectively generate audio adversarial examples.

heading: 2 RELATED WORK
text: In ASR adversarial attacks, Carlini & Wagner (2018) were among the first ones to showcase that slight perturbations on audio can easily fool a start-of-the-art ASR model to transcribe the perturbed audio into any target sentence. Later on, Yuan et al. (2018) demonstrated that such adversarial audio perturbation can be embedded into a song. Yang et al. (2019) analyzed that the temporal dependency could promote the discriminative power against adversarial examples in ASR. Moreover, Khare et al.
(2019) and Taori et al. (2019) found that similar audio adversarial perturbations could be generated using black-box optimization algorithms (e.g., genetic algorithm (Mitchell, 1998)). To generate the audio adversarial examples faster and improve the attack efficiency, Liu et al. (2020) proposed to adaptively rectify the weights of audio perturbations on different positions. To make the audio perturbation more imperceptible, Qin et al. (2019) introduced a psychoacoustic principle of auditory masking to smartly add perturbations on frames where noises are hard to be perceived. Xie et al. (2021) utilized generative adversarial network (GAN) to generate adversarial examples on speech domain, where GAN is to learn the distribution of predefined adversarial perturbations. Thus the adversarial examples are still generated depending on the benign audios.
In summary, previous adversarial attacks on ASR are audio-dependent, viz., the audio adversarial examples must be generated based on corresponding benign audios. Although there are studies, e.g., (Song et al., 2018; Wang et al., 2019), on generating adversarial examples from scratch, but most of them focus on the image domain. Carlini & Wagner (2018) studied an audio attack that starts from non-speech (e.g., a piece of classic music), while existing audios are stilled required to mount perturbations on. Roy et al. (2018) and Zhang et al. (2017) proposed to modulate voice commands on ultrasonic carriers (e.g., frequency > 20 kHz) to achieve inaudibility, where the scenario is an adversary that stands on the road and silently controls the voice command assistant systems. This therefore is different from our scope of generating natural sounding adversarial audios. To the best our knowledge, our work is the first attempt to generate audio adversarial examples from scratch without utilizing benign audios in ASR.

heading: 3 BACKGROUND
text: Automatic Speech Recognition (ASR). In this paper, we focus on the speech-to-text tasks that are based on neural ASR models. Following previous studies (Esmaeilpour et al., 2021; Liu et al., 2020; Yakura & Sakuma, 2019; Carlini & Wagner, 2018), we stick our attacks on the DeepSpeech (Amodei et al., 2016), a state-of-the-art ASR model based on Connectionist Temporal Classification (CTC) method (Graves et al., 2006). DeepSpeech uses CTC as the input aduios and the corresponding transcriptions are unaligned, viz., the exact position of each word in the audio sample is unknown. To enable efficient supervised training, a transcription is first enumerated to obtain all alignments. Hence, the CTC loss is minimized to maximize the probabilities over all alignments.
CTC Loss. Let X be the audio input domain and Y be the text output domain with dimension |Y|. The ASR model is donoted as F : XN → YN ·|Y|, which takes a N frames x ∈ X as input and outputs a probability distribution F(x) over the output domain. Given y as a phrase (i.e., a sequence of characters), we define a token sequence π being reducible to y if the two operations, namely, removing sequentially duplicated tokens, and deleting all blank tokens on π could produce y. We further denote π as an alignment of y if π can be reduced to y and the length of π equals to the length of prediction F(x). Let Φ(y) be the alignment set obtained from targeted transcription y using dynamic programming (Graves et al., 2006). Accordingly, the CTC loss can be formulated as,
Lctc(x, y) = −log [∑
π∈Φ(y) ∏N i F(x)iπi ] , (1)
whereF(x)iπi is the probability of the token πi on the ith frame. To get the transcription in inference, a decoder D (e.g., greedy decoding or beam search decoding) is required. Thereby, if an ASR model is well trained, the transcription would satisfy y = D(F(x)) = f(x), where f(·) is a merged denotation of F and D. White-Box Assumption. Following most of previous studies, we use the similar white-box assumption, viz., the parameter and structure of ASR model is known. Investigating the black-box attack is another direction. Moreover, there are many ways (Oh et al., 2019; Zanella-Beguelin et al., 2021; Qin et al., 2019) to convert a black-box model to a white-box one.
Targeted Attack. Compared with the untargeted attack that only maximizes the word error rate (WER), the targeted attack is a more challenging task since it requires not only the audio perturbation imperceptible, but also the adversarial audio being transcribed to a specified target phrase. Our main focus is on the targeted scenario. Without specification, the adversarial example in following sections refers to the targeted one.
Speech Synthesise (End-To-End Text-To-Speech). Text-to-speech (TTS) model synthesises waveforms given text phrases as semantic contents. For efficiently synthesising adversarial audios in our SSA, we turn to the end-to-end TTS model that could easily utilize modern parallel processors for faster synthesis speed. In particular, we utilize a conditional variational autoencoder (Kim et al., 2021) based TTS model to generate natural sounding audios, which is explained in Section 4.2.

heading: 4 METHODS

heading: 4.1 PROBLEM SETTINGS
text: Given a well trained ASR model f(·), the objective of adversarial attack is to construct an audio waveform x ∈ X that is naturally sounded yet able to deceive f(·) in predicting incorrect/targeted transcriptions. Suppose o : X → Y is an oracle that takes an audio waveform x as input and outputs the ground truth transcription yo = o(x), where Y is the set of all text transcriptions under consideration. Moreover, compared to the untargeted attack that only introduces spelling errors, we focus on the more challenging targeted attack, viz., f(x) = y where y ∈ Y is the expected target transcription by an attacker.
In previous studies, the adversarial audio x is constructed by adding perturbation δ on a benign audio xo, viz., x = xo + δ; thus being dependent on xo. Mounted on these notations, we give a formal definition of the previous audio dependent attack as follows. Definition 1 (Audio Dependent Attack - ADA). Given a benign audio xo and its oracle transcription yo = o(xo), the corresponding adversarial example can be defined as any audio x, viz., x ∈ Aδ ∆ = {xo + δ ∈ X |∃δ,M(δ) ≤ ϵ︸ ︷︷ ︸∩ o(xo + δ) = yo = o(xo)︸ ︷︷ ︸∩ f(xo + δ) = yt︸ ︷︷ ︸∩ f(xo) ̸= yt︸ ︷︷ ︸∩ yo ̸= yt︸ ︷︷ ︸}, where M(·) is a distance measurement (e.g., matrix norm); ϵ is a small positive constant; and yt indicates the targeted transcription of an attacker.
From Definition 1 the adversarial audio x is directly built on a benign audio xo. Moreover, to guarantee x being acoustically realistic or natural, the efforts mainly focus on forcing M(δ) ≤ ϵ with the goal of restricting x to be close to xo. However, in some cases, the benign audio xo may not be available. For instance, an attacker want to deceive a voice commander when no human speaking happens nearby. Moreover, ADA relies on an imperceptible perturbation principle, viz., the added perturbation δ must be small enough to avoid being percepted by human beings. In contrast, a more general scenario would be audio independent attack that is defined as follows. Definition 2 (Audio Independent Attack - AIA). Given a conditional text (i.e., yo), an audio independent adversarial example can be any element from Aa ∆ = {x ∈ X | o(x) = yo︸ ︷︷ ︸∩ f(x) = yt︸ ︷︷ ︸∩ yo ̸= yt︸ ︷︷ ︸}, where o(x) = yo indicates that the synthesised audio x correctly conveys its semantic content yo; and f(x) = yt means the ASR model f(·) is successfully fooled to output the targeted transcription yt.
Such AIA sheds light on a more general principle of adversarial attacks, viz., any audio that deceives ASR models yet fails to deceive humans would cause security issues in speech recognition. In other words, the adversarial audios are not necessarily constructed via adding perturbations. Instead, they can be directly synthesized with the goal of preserving the desired semantic content (i.e. o(x) = yo), while simultaneously deceiving the ASR model to predict incorrect or even targeted transcriptions (i.e. f(x) = yt). This motivates us to leverage the powerful generative model in TTS area to construct such adversarial audio x. Next, we will introduce the speech synthesising based attack.

heading: 4.2 SPEECH SYNTHESISING BASED ATTACK
text: The overall structure of the speech synthesis based attack (SSA) is depicted in Figure 2.
Conditional Variational Autoencoder based Speech Synthesis. The key for synthesising natural sounding adversarial audio is to model the TTS mapping. In practice, we can select different types of TTS models (Tan et al., 2021) to generate natural sounding audios. We choose the recent conditional variational autoencoder (CVAE) based TTS model (Kim et al., 2021) due to its rich variety in audio generation, viz., a text input can be spoken in multiple ways with different pitches and rhythms.
Algorithm 1: SSA Algorithm Input: yo – the ground truth text; yt – the target for attack; G(·) – the generative model for speech
synthesise; F(·) – the ASR model; α0 – the initial learning rate; dα – the learning rate decay ratio; λ – the weight of Lreg(z); Im – the maximum iteration number; Dls(·) – Levenshtein distance.
1 Initialization: audio style vector z ∼ N (0,1); the patience p = 0; record best loss L∗ = +∞; 2 for i ∈ N+ ∩ i < Im do 3 Calculate the SSA loss Lssa(z, yo, yt) in Eq. (3); 4 if Lssa(z, yo, yt) <= L∗ then 5 L∗ = Lssa(z, yo, yt), p = 0 6 else 7 p← p+ 1 8 if p >= pm then 9 α← dα · α, p = 0
10 Do back-propogation to get ∂Lssa(z,yo,yt) ∂z
and update z following Eq. (5); 11 if Dls(f(G(z, yo)), yt) == 0 then 12 Return the successful audio adversarial example x∗ = G(z, yo)
13 Return the current best audio adversarial example x = G(z, yo)
This accordingly enlarges the sample space, and enhances the possibility of constructing successful adversarial examples. In specific, such CVAE based TTS model (Kim et al., 2021) is based on three components: 1) a conditional VAE formulation; 2) an alignment estimation derived from variational inference; 3) an adversarial training for improving the synthesis quality. The architecture of the CVAE model in inference is shown in Figure 2, where the corresponding training loss and settings can refer to (Kim et al., 2021).
In particular, given a conditional text ctext = yo (where yo conveys the ground truth semantic content), the TTS model aims to build the mapping G(·), viz.,
x = G(z, ctext), z ∼ N (0,1), (2)
where z is a normally distributed vector that controls the audio styles with different pitches and rhythms. Therefore, with different z, there are different ways to speak the content in ctext. This aligns to the fact, viz., even human beings always pronounce same words differently from time to time. Note that the audio x generated by CVAE model G(·) in (Kim et al., 2021) has been tested to be natural sounding using the mean opinion score obtained from Amazon Mechanical Turk (www.mturk.com). The following goal thus becomes figuring out a particular z that can fool an ASR model F(·). Speech Synthesising based Attack (SSA) Formulation. We focus on the targeted attack, viz., deceiving the ASR modelF(·) to predict a target phrase yt. Given the speech synthesiser G(z, ctext) in Eq. (2), our SSA is formulated as finding a particular z∗ during synthesising audio x that can enable the targeted attack. To this end, a loss function Lssa is designed to construct natural sounding yet fooling enough x. Accordingly, the optimization of z can be defined as
z∗ = argmin z Lssa(z, yo, yt) = argmin z Lctc(G(z, yo), yt) + λLreg(z), (3)
where yo is the phrase that the CVAE model wants to synthesise; yt is the targeted phrase that the ASR model is fooled to predict; and Lreg(z) is the regularization loss controlled by λ. To be specific, Lreg(z) is designed to boost the generated audio to be naturally sounded. Based on (Kim et al., 2021), z is sampled from a normal distribution. To preserve this property, we design it as:
Lreg(z) = ϕ(µ(z)) + ϕ(δ(z)− 1), (4)
where ϕ(·) is an absolute value function; µ(z) and δ(z) are the mean and variance of z, respectively. To detemine whether the generated audio x = G(z, yo) can enable a successful targeted attack (i.e., f(x) == yt), we involve the Levenshtein distance (Yujian & Bo, 2007). During the optimization of Lssa(z, yo, yt), if Dls(f(G(z, yo)), yt) = 0, the targeted attack is successful. Adaptive Sign Gradient Descent. To solve the optimization problem in Eq. (3), we propose an adaptive sign gradient descent algorithm whereby an adaptive learning rate decay mechanism is
designed in the sign gradient descent optimization. Typically, the sign gradient descent is frequently adopted in adversarial attack algorithms, e.g., fast gradient sign method (Goodfellow et al., 2015) and projected gradient descent (Madry et al., 2018). Moreover, Balles & Hennig (2018) theoretically proved the benefits of using gradient sign as the optimization direction. In our SSA, the sign gradient descent based optimization is given by,
z ← z + α∂Lssa(z, yo, yt) ∂z , (5)
where α is the learning rate, which significantly impacts the convergence of the SSA optimization. Inspired by the heuristics of annealing (Bertsimas & Tsitsiklis, 1993), we carefully design an adaptive learning rate decay mechanism as below. Definition 3 (Annealing based adaptive learning rate decay). For every pm steps where the attack loss Lssa(z, yo, yt) has no improvement, the learning rate α will decay as α := dα · α, where dα ∈ [0, 1] is the decay ratio.
Definition 3 tells that if a specific α gets stuck for particular pm steps with regard to Lssa(·), α should decay to perform a more local search. The overall process of SSA is shown in Algorithm 1.

heading: 5 EXPERIMENTS AND RESULTS

heading: 5.1 EXPERIMENT SETTINGS
text: Datasets. In our experiments1, we use three datasets, i.e., Audio Mnist (Becker et al., 2018), Common Voice (Ardila et al., 2019), and Librispeech (Panayotov et al., 2015). However, different from previous studies using both the waveform and text label, we only utilize the text information due to the audio independent property of our SSA. In particular, the Audio Mnist contains the text digits (i.e., from “ZERO” to ”NINE”). When building the targeted attack pairs, we choose one text digit (e.g.,“ZERO”) as yo and enumerate the rest digits (i.e., from “ONE” to “NINE”) as its attack target yt. For Common Voice and Librispeech, we randomly sample 1000 text labels first. Those sampled texts are filtered and clustered based on their text length. Thereafter, 100 texts are sampled out from the filtered 1000 samples as the candidates of conditional text yo. The corresponding target text yt is sampled from their matching clusters. The length comparison between yo and yt on the two datasets are shown in Figures (10-11) in the appendix, where we notice that yo and yt are aligned with a similar length. More analysis can refer to appendix 7.1. Our constructed datasets2 are released to benefit future research on speech synthesise/generative model related attacks.
Model Settings. In our SSA paradigm, the CVAE model and DeepSpeech are utilized as speech synthesiser and speech recogniser, respectively. The CVAE model follows the setting of VITS (Kim et al., 2021) during inference. In particular, we utilize the CVAE model trained on VCTK dataset (Veaux et al., 2017), which can be obtained from VITS3. The standard deviation of the input noise for the stochastic duration predictor is set to be 0, thus making it a deterministic one. In addition, a scaling factor for z is applied. The DeepSpeech model follows the Pytorch implementation in the adversarial robustness toolbox (Nicolae et al., 2018).
Evaluation Metrics. Two evaluation metrics, i.e., the word error rate (WER) and success rate (SR), are adopted under the targeted attack setting. In particular, they are defined as,
WER = S +D + I
Nw , SR = Ns Na
(6)
where S,D and I indicate the number of subsitutions, deletions and insertions of words respectively; Nw is the total number of words; Ns is the number of successful adversarial example (i.e., f(x) = yt); and Na is the total number of audios synthesised. Note that SR is same as the sentence-level accuracy that is used in previous studies (Qin et al., 2019). In general, larger values of WER and SR indicate a stronger attack algorithm.
SSA Optimization Settings. The learning rate α decays in updating z as highlighted in Definition 3 and Algorithm 1. In particular, α0 (i.e., initialization of learning rate) is searched in the range of
1Our code will be released upon acceptance. 2 https://drive.google.com/file/d/1EHXRlWrlMXr6qu8WYtjt8-pRzw-VfCau/view?usp=sharing 3 https://github.com/jaywalnut310/vits
[0.01, 0.09] stepped by 0.01; the patience pm ranges from 50 to 400 stepped by 50; the learning rate decay ratio dα is searched in {0, 5, 0.6, 0.7}; moreover, the regularization weight λ is searched in {0, 50, 100, 150, 200}. The maximum iteration number Im is 8000, where any case without finding a successful attack within Im budget is deemed as failed.

heading: 5.2 TARGETED ATTACK PERFORMANCE
text: We first show that our proposed SSA is capable of efficiently constructing audio adversarial examples, viz., significantly outperforming existing baselines in Table 1. In particular, the baselines include DeepSpeech4, C&W attack (Carlini & Wagner, 2018), Y&S attack (Yakura & Sakuma, 2019), GAA attack (Taori et al., 2019), MOOA attach (Khare et al., 2019), Metamorph attack (Chen et al., 2020), and CIPMA attack (Esmaeilpour et al., 2021). The results from the baselines are reported in (Esmaeilpour et al., 2021), which are averaged results over Common Voice and Librispeech. Our SSA is evaluated on three datasets (i.e., Audio Mnist, Common Voice, and Librispeech), where an averaged result over Common Voice and Librispeech is calculated for a fair comparison. All the results of our SSAs are based on targeted attack settings.
In general, we can see that our SSA achieves remarkably better performance on both WER and SR. Specifically, (1) SSA-Average achieves 103.49% and 89.60% regarding WER and SR, respectively, which outperforms the best results obtained by CIPMA on WER (i.e. 88.19%) and GAA on SR (i.e. 48.35%); (2) both SSA-Common Voice and SSA-Librispeech dramatically outperform the best results obtained from the baselines as well. Especially for SSA-Common Voice, both of its WER (i.e. 106.27%) and SR (i.e. 96.00%) are the best across all comparisons; (3) SSA-Librispeech behaves worse than SSA-Common Voice with regards to WER and SR, which is mainly attributed to the difference of conditional text lengths between the two datasets; and (4) our proposed SSA realizes 100% w.r.t. both WER and SR on Audio Mnist, which can serve as a strong baseline for future study on the audio adversarial attack. We suggest the reader to listen to our synthesised adversarial audios that are available at our webpage5.

heading: 5.3 QUALITY EVALUATION OF SYNTHESISED AUDIOS
text: Although the regularization loss Lreg(z) in our SSA is designed to force our synthesised audios to be independently and identically distributed w.r.t. the audio generated in VITS [1], we still find some synthesised audios not natural sounding enough. Therefore, we further evaluate the quality of synthesised adversarial audios by mean opinion score (MOS) tests. Specifically, we invited 20 participants to rate on 50 sample pairs, where each sample pair includes an original synthesised audio by CVAE and its corresponding synthesised adversarial audio optimized by our SSA. Each participant will listen to these 100 audio samples that are randomly shuffled. The naturalness rating
4 https://github.com/Picovoice/speech-to-text-benchmark#mozilla-deepspeech 5 https://sites.google.com/view/ssa-asr/home
score is scaled from 1 to 5. The MOS results are shown in Table 2, where we can observe a slightly worse MOS score of our SSA synthesised audios (i.e., 3.39 ± 0.29) compared with that of original synthesised ones (i.e., 4.09 ± 0.10). This again indicates that there exists distortions in the synthesised audios by SSA. However, we further note that the difference of MOS scores between the two types of synthesised audios is not significant, which indicates that the distortions are still subjectively acceptable. Future works can focus on how to eliminate such distortions in the adversarial audio generation, such as redesigning the regularization loss Lreg(z) in Eq (3).

heading: 5.4 HYPERPARAMETER ANALYSIS ON THE ADAPTIVE SIGN GRADIENT DECENT
text: In the adaptive sign gradient decent, the initial learning rate α0 and the patience pm are two important hyperparameters. Their impacts regarding SR on Audio Mnist and Common Voice datasets are displayed in Figure 3 and 4, respectively. Moreover, we also analyze their impacts on Common Voice w.r.t. WER. Several interesting findings are noted as below.
Analysis on SR. (1) Generally, the SR first climbs up as α0 increases and then keeps stable with further increase of α0. For instance, a stable SR is achieve with α = 0.03 on Audio Mnist. Similar trends are also held with pm. (2) On both Audio Mnist and Common Voice, pm needs to be set as a mild value (e.g., pm ∈ {300, 350}) in order to obtain a promising SR. This indicates that if the learning rate α decays too fast, the sign gradient descent optimization in Algorithm 1 may get stuck. (3) Compared the results in Figures (3-4), the SR from Common Voice is usually smaller than that from Audio Mnist even with the same settings on α0 and pm. This is mainly because the audio synthesised in Common Voice is clearly longer than that from Audio Mnist.
Analysis on WER. Figure 5 shows the WER with different settings of α0 and pm. As a whole, the WER first goes up with the increasing of both parameters, while becomes less sensitive as they continue to increase. For example, comparing across different lines (i.e., with different α0), we can only ob-
serve a slight change on WER with α0 ≥ 0.04. This leads to a similar conclusion with the analysis on SR, viz., proper settings of α0 and pm can easily synthesise harmful adversarial audios via SSA.

heading: 5.5 CONVERGENCE ANALYSIS
text: One natural question is, given a particular setting on α0 and pm, how does the process of adaptive sign gradient descent look like? We choose the best setting according to the results in Figure 4, viz., α0 = 0.06 and pm = 400, and select six representative evaluated cases to show the convergence of the CTC loss, Levenshtein distance on Common Voice in Figures 6(a-b), respectively. In addition, to assist the analysis, the correspond text length comparison and learning rate decays are provided in Figures 6(c-d). Due to space limitation, the overall convergence process across 100 samples related to CTC loss and Levenshtein distance and learning rate decays are respectively shown in Figures (12-14) in the appendix. Some interesting observations are noted as follows.
First, from Figure 6(a) and Figure 12, we can see that the CTC loss of most cases quickly converges to a small value close to 0. The convergence speed is especially fast at the beginning of the optimization. Such a phenomenon is also observed in other studies (Amodei et al., 2016) related to CTC loss based training. Second, in Figure 6(b), the Levenshtein distance on most cases except case 1 converges to 0, indicating a successful targeted attack. Third, on some cases, e.g., case 1 in Figure 6(a), although the convergence curve does not stop before reaching the maximum iteration step 8000, it can be deemed as an approximately successful attack as reflected by the small CTC loss in Figure 6(a) and Levenshtein distance in Figure 6(b). Fourth, the additional text length comparison in Figure 6(c) shows that the text length generally reflects the difficulty of generating a successful attack, viz., a longer text usually requires more iterations in both CTC loss and Levenshtein distance convergences. Lastly, the learning rate dynamics in Figure 6(d) also showcase that for a harder problem (i.e., with a larger text length), the learning rate decays more times to exploit a solution close to a successful attack. More results and analyses can refer to Appendix 7.2.

heading: 5.6 WAVEFORM PATTERN ANALYSIS
text: To showcase that our SSA is a more general attack compared with audio dependent attacks, we further analyse the waveform pattern of both attacks as shown in Figure 7. In particular, Figure 7(a) and (b) respectively depict the original audio and the corresponding adversarially perturbed audio, where we can easily observe that the attacked audio needs to be restricted to only add minor perturbations. In contrast, the adversarial audio constructed by our SSA as shown in Figure 7(c) is free of such restriction, viz., the waveform can be significantly different. In sum, our SSA can be deemed as an audio independent attack, which brings in more threat to ASR models in the wild. Further analyses towards the audio style vector z before and after attack of SSA are shown in Appendix 7.3. In addition, we also evaluate the tranferability of the adversarial audios w.r.t another ASR model (i.e., ESPnet (Watanabe et al., 2018)) as shown in Appendix 7.4.

heading: 6 CONCLUSION
text: This paper investigates the audio adversarial attack for ASR models. Existing attack algorithms are based on an audio dependent assumption, viz., adding constrained perturbations on benign audio inputs. In contrast, we propose SSA, a novel threat model that constructs audio adversarial examples entirely from scratch, viz, without depending on any existing audio to fool cutting-edge ASR models. To this end, we propose to use a conditional variational auto-encoder (CVAE) as the speech synthesiser. Accordingly, the adversarial audio synthesising task is formulated as an optimization problem via searching in the hidden space of CVAE. Meanwhile, an adaptive sign gradient descent algorithm is further devised to solve the SSA optimization problem. Experiments on three datasets show that our proposed SSA can synthesise audios that are naturally sounded but deceive start-ofthe-art ASR models. In our experiments, we also find that some synthesised adversarial audios do not sound as natural as those without any manipulation on z, which thus needs future efforts to enhance the quality of adversarial audio synthesis.

heading: 7 APPENDIX

heading: 7.1 DATASET ANALYSIS
text: Figure 9 shows the length distribution of 1000 samples on Common Voice and Librispeech. Note that, to better sample the adversarial attack target, we filter out data points that have too small counts. From the selected 1000 samples, we generate our targeted attack dataset, viz., matching a target that has a similar length with the conditional text. Figures (10-11) illustrate the comparison of the text length between conditional text yo and target text yt on Common Voice and Librispeech, respectively. In general, the length of yo and yt are similar. Moreover, in both datasets, the text length has a huge variance, e.g., case 3 versus 30 in Librispeech. Compared Common Voice to Librispeech, the length of many cases on Librispeech is much larger than the maximum length (i.e., 100) on Common Voice.

heading: 7.2 ADDITIONAL ANALYSIS ON THE CONVERGENCE OF CTC LOSS AND LEVENSHTEIN DISTANCE AND LEARNING RATE DECAY
text: The CTC loss convergence processes across the 100 samples on Common Voice are shown in Figure 12. Moreover, we further analyse the convergence of Levenshtein distance Dls(·) (Khare et al., 2019) in Figure 13, in order to depict how does our proposed SSA succeeds. Note that the Levenshtein distance is calculated as Dls(f(G(z, yo)), yt), which indicates the distance between the transcription on the current synthesised audio x and the target transcription yt. We also analyse the dynamics of α in Figure 14.
From Figure 12, we can see that the CTC loss of most cases quickly converges to a value close to 0. The corresponding Levenshtein distance in Figure 13 exactly converges to 0 on these cases, suggesting successful attacks on most cases. For the cases that do not stop the optimization before reaching the maximum iteration step 8000, both the CTC loss and Levenshtein distance end with small values, which can be deemed as approximately successful attacks based on the related studies (Zhang et al., 2020) on adversarial attack in natural language processing. From Figure 14, generally different cases have their own learning rate schedule. On most cases, the learning rate needs to decay for at least 2 times. Only on some particular cases (e.g., case 2), directly using the initialized α without decay can find a successful adversarial example. This implies that our designed adaptive sign gradient descent algorithm enables the learning rate α to dynamically change according to the optimized loss.
7.3 ANALYSES ON THE AUDIO STYLE VECTOR z
We further compare the original audio style vector z (i.e., sampled from the normal speech synthesis) and the adversarial z (i.e., optimized by our SSA loss) on three cases as shown in Figure 8. For better visualization, we only plot partial of vector z. For instance, z is reshaped from a (192× 231) matrix with dimension determined by the conditional text in Figure 2, while we only plot (2 × 231) of them as shown in Figure 8 (a). In general, Figure 8 shows that the original z and adversarial z are significantly different. Namely, although both original and adversarial z fluctuate around the mean 0 and share a comparable variance, their specific values for each dimension are quite different. This indicates that our SSA has more flexibility of searching for a successful attack. In contrast, the optimization space of previous audio dependent attacks is restricted around the original audio waveform by a norm bound as shown in Figure 7 (b).

heading: 7.4 ANALYSES OF ATTACK TRANSFER
text: Our SSA is designed to be ASR model dependent. In specific, the adversarial audios are synthesized based on Deep Speech. Therefore, as expected, these adversarial examples should pose limited threat to other ASR models. To validate such a hypothesis, we mount the successfully synthesised audio attacks on ESPnet (Watanabe et al., 2018) (i.e., an attention-based encoder-decoder network). In doing so, we randomly sample 30 successfully synthesised attacks (i.e., based on Deep Speech), input them to the ESPnet and calculate the levenstein distance (LD) with respect to the target text, where LD = 0 indicates a successful targeted attack. Results show that the success rate and LD of
these transferred attacks on ESPnet are 0% and 40.97 ± 15.34, respectively. This suggests that the adversarial audios generated by SSA are hardly transferable to a different ASR model.

heading: 7.5 HUMAN SPEECH RECOGNITION (SR) EVALUATION
text: The human speech recognition (SR) evaluation has been conducted. Specifically, we invite 5 participants to listen to 50 sample pairs, where each sample pair includes an original synthesised audio by CVAE and its corresponding synthesised adversarial audio optimized by our SSA. Each participant then writes down the corresponding translation text. The WER is calculated between the human translation and ground truth text. The averaged WER from the human evaluation is shown in Table 3, which indicates that the human translation performance is only slightly impacted compared to the original synthesised audios.

Review summaries:
This paper presents a speech synthesising-based attack (SSA) framework that generates audio examples entirely from scratch to attack automatic speech recognition (ASR) systems. A conditional variational auto-encoder (CVAE) based speech synthesiser is trained based on a combined connectionist temporal classification (CTC) and regularization loss. As compared to existing methods, the proposed approach serves as the first work that generates adversarial samples without audio inputs. The results confirm that the proposed SSA framework can attack the ASR system successfully. A comprehensive analysis of different hyper-parameters to the achievable performance is provided. Moreover, demo audio samples are provided by a website link.  

This paper describes a white-box attack on ASR systems using TTS.  The main technique is (somewhat) standard adversarial training with the constraint that the CVAE component of a TTS model is the only component that is updated.  Since this component is responsible for prosodic and other non-segmental qualities of speech synthesis, the expectation is that the naturalness of the synthesis will not be impacted, but the ASR recognition will be modified.

The paper presents a new adversarial attack mode to ASR systems based on speech synthesis. The main idea is to synthesize speech utterances that naturally sound like ground-truth transcription y_o, but can fool an ASR system. It is a targeted attack model, where each attack signal has its own target label y_t. The paper also presents a CVAE-based speech synthesis system and a learning rate decay method for training. 

In this work the authors explore a new vein of research in the area of audio adversarial attacks wherein instead of focusing on perturbing existing audio, they synthesize audio. To their best knowledge, and this reviewer, this is  the first such attempt. To this end, the authors use a conditional variational auto-encoder as a speech synthesis (TTS) model and develop  a adaptive sign gradient decent algorithm in order to solve their SSA optimization problem. Experiments are provided showing the effectiveness of  their approach, along with a detailed analysis of results.