Demystifying Limited Adversarial Transferability in Automatic Speech Recognition Systems
Authors: Hadi Abdullah
Aditya Karlekar
Vincent Bindschaedler
Patrick Traynor
Venue: ICLR 2022 Poster (2022)
Keywords: optimization attacks
transferability
adversarial machine learning

Abstract: The targeted transferability of adversarial samples enables attackers to exploit black-box models in the real-world. The most popular method to produce these adversarial samples is optimization attacks, which have been shown to achieve a high level of transferability in some domains. However, recent research has demonstrated that these attack samples fail to transfer when applied to Automatic Speech Recognition Systems (ASRs). In this paper, we investigate factors preventing this transferability via exhaustive experimentation. To do so, we perform an ablation study on each stage of the ASR pipeline. We discover and quantify six factors (i.e., input type, MFCC, RNN, output type, and vocabulary and sequence sizes) that impact the targeted transferability of optimization attacks against ASRs. Future research can leverage our findings to build ASRs that are more robust to other transferable attack types (e.g., signal processing attacks), or to modify architectures in other domains to reduce their exposure to targeted transferability of optimization attacks.

DEMYSTIFYING LIMITED ADVERSARIAL TRANSFER- ABILITY IN AUTOMATIC SPEECH RECOGNITION SYS-
TEMS
Hadi Abdullah, Aditya Karlekar, Vincent Bindschaedler, & Patrick Traynor University of Florida
(hadi10102, akarlekar, http:/........aakekkoooaaaaaaeed vbindschaedler, traynor)@ufl.
ABSTRACT
The targeted transferabilityy of adversarial samples enables attackers to ex- ploit black-box models the real world. Optimization attacks are the most popular means of producing such transferable samples. This is be- cause these samples have high levels of transferability in some domains. However, recent research has shown that samples from these attacks do not transfer when applied to Automatic Speech Recognition systems (ASRs). In this paper, study this phenomenon, perform exhaustive experiments, and identify the factors that are preventing transferability in ASRs. To do so, perform ablation study cach stage of the ASR pipcline. We dis- cover and quantify six factors (i.e., input type, MFCC, RNN, output type, and vocabulary and sequence sizes) that impact the largeted transferabil- ity of optimization attacks against ASRs. Our findings can be leveraged to design ASRs that are more robust to other transferable attack types (e.g.. signal processing attacks), or to modify architectures in other domains to reduce their vulnerability to targeted transferability.
INTRODUCTION
It is hard to understate the pervasiveness of ASRs in security-critical systems. These in- clude banking (Amazon) 2021]. surveillance Froomkin 2015). online retail (Vigderman 2021), and home security (Bharadwaj) 2019). However, ASRs are vulnerable specially crafted adversarial samples, that force them to produce malicious outputs. In the research community, the most popular class of algorithms for this purpose are the optimization (or gradient-based) attacks (Abdullah et al. 2021b). This is largely because these attacks ex- hibit targeted transferubility in some domains. Specifically, samples crafted for a local model (surrogate) can force different black-box model (target) to produce the attacker chosen output. This the case even if the surrogate and target use different architectures, training data, prper-parameters etc. Therefore, the transferability property has enabled attackers lo exploit security-critical applications which include facial recognition systems (Shan el al. 20201, image APIs (Brown al. 2017 Liu al.) 2016), authentication systems (Chen et al.
2019 and malware detectors (Hu & Tan 2017 Kreuk et al. 2018b Grosse et al., 2017). In short, target transferability makes optimization attacks eftective against several real-world systems.
However, recent work has uncovered that these attacks do not exhibit target transferability between ASRs, even when the ASRs are trained on identicol setups (i.c., same hyper- parameters, architecture, random seed, training data) (Abdullah et al., (0021)) Therefore, these aacks can not be used against black-box ASRs, casting doubt. on the usefulness of the entire class of optimization attacks the audio domain. However, the reasons for this failure is not understood.
In this work, we study this phenomenon to uncover the factors that prevent the transfer- ability of optimization attacks between ASRs. To ensure factors, we perform an exhaustive ablation study on the entire ASR pipeline and observe the impact of the different components the transferability rate. We test thousands of adversarial samples across multiple models and characterize six factors that impact the transferability rate. In doing so, make the following contributions:
uncover each one of these

---

1. We identify six previously unknown factors that impact target transferability. These include the imput type, Mel Frequency Cepstral Coefficient (MFCC), the Recurrent Neural Network (RNN), output type, and the vocabulary and sequence sizes. These factors explain the near 0% transferability rate seen in prior works.
2. We highlight the relationship between accuracy and adversarial robustness in ASRs. The five factors that improve ASR robustness also improve accuracy.
3. Our findings explain why one of the most popular classes of attacks, across the ad- versarial machine learning space, fails in the audio domain. These can be leveraged strengthen models from other domains (e.g., images), that have traditionally been vulnerable to optimization attacks.
We begin our study by listing all the factors from the existing literature that are known to hinder transferability (Section 2.1 Even when controlling for these factors, transferability rates in ASRs still do not achieve the near 100% observed in the image models (Section 4.1 This suggests the existence additional factors limiting transferability. We list the addi- tional potential factors (Section 2.3], describe our design choices (Section 3), and through a series of ablation experiments, we systematically quantify the impact each factor on transfcrability (Section[]) 4h. Based our findings, we discuss several takcaways (Section present related work (Section and summarize our findings (Section
2
FACTORS
2.1
KNOWN FACTORS FROM EXISTING LITERATURE
Before delving into ASRs, it is first important to review the existing literature on trans- ferability. While doing so, we identified 11 factors that are already known to limit transfer- ability of optimization attacks:
1. Smoothness of gradients (Demontis et al. 2019
Zhou et al
2018
Wu et al. 2018
2. Attack type (Kurakin et al. 2016a Dong et al.
2018
Liu et al.
2016
3. Number of attack iterations Dong
2018
4. Number of output labels (Liu et al.
2016
5. Spectral makeup of the perlurbations Shaarma el al.
2019 Guo el al. 2018
6. Model architecture (Wu et al. 2018
7. Model accuracy (Wu et al. 2018)
8. Model complexity (Demontis et al.
2019
Wu al.
2018 Wu & Zhu
2020
9. Model agreement (Tramer et al.
2017)
10. Confidence of the adversarial sample Abbdullah et al. 2021b
11. Asymmetry (Wu et al. 2018 Wu & Zhu 2020
2021b), the primary focus of most of these works has been image classification models. As a consequence, even when controlling for these factors, we observed that transferability between ASRs remains low (Section 4.1h. This suggests the existence of unknown factors limiting transferability.
Unfortunately, outside of a single example (Abdullah et al.
2.2 ASR PIPELINE
To identify these unknown factors, we first provide brief overview of the different com- ponents that make up the ASR pipeline (Figure []. To that end, we consider the most commonly attacked ASR pipeline in the research community (Abdullah et al. 2021
The first stage of an ASR splits the input audio into overlapping frames (Figure 11)). Next, a signal processing algorithm, (e.g., the MFCC (Lin & Abdulla 2015 extracts feature vector from each of the overlapping frames (Figure (i)) Next, the neural network (Figure (iii) assigns a single character label to each feature vector, resulting in a character list (c.g., "hheellllo bbbboooob") (Figure [iv).. These are then aggregated into single word (e.g.. "hello"), which is then combined with other words into a final sequence (c.g. "hello bob") (Figure [v)).
2

---

Potential Factors
Input Type
bJMECC
c) ROON
d) Output type
Vocabulary Sequence size
MPCC
CNN
n. 1
bio
Stages
Audioo
(18) giaal Processiny
(1 Neural Setwork
Output
Ivi Label Agginggati
Framss
Figure 1: The typical ASR pipeline used in the adversarial research community. We can see the potential factors (a-f) and their locations different stages (i-v) of the pipeline. We will study cach of the impact of cach potential factor on transfcrability.
2.3 ADDITIONAL POTENTIAL FACTORS
Having described the ASR pipeline, we can now list the components that might potentially impact transferability. We will later experimentally investigate their impact during ablation study. These potential factors include:
2.3.1 INPUT TYPE:
Given previous findings (Nascer et al. 2019 we hypothesize that the model's input type can impact transfcrability. For example, audio inputs are time-varying and one-dimensional. They do not contain topographical structures. In contrast, images are two-dimensional, topographical inputs with neighborhoods of pixels representing single object (e.g., pixels representing dog's snout). As result, the input type will determine the model's decision boundaries, which can impact transferability.
2.3.2 MFCC:
We hypothesize that the MFCC algorithm could be factor limiting transferability. This algorithm uses hand-crafted filters to regularize the feature vector and consequently, remove the high spectral noise (Mannell 1994). Since regularization of the decision boundary or gradients has been shown to impact transferability (Demontis et al. 2019 Zhou et al. 2018
Wu al. 2018 Wu & Zhu 2020), we hypothesize that regularization of the feature vector using the MFCC plays a role as well.
2.3.3 RNN:
The network architecture impact (ransferability as well. For example, the dropout layer, which is often included in model architectures for performance purposes, also inadvertently improves the transferability of adversarial samples (Demontis al. 2019).
We hypothesize that the RNNis a factor limiting transferability in ASRs. This is because there are several RNN-based applications where authors have failed demonstrate targeted transferability (Abdullah et al. 2021b). While these papers show transfcrability is hard in their respective domains, they do not contain definitive experiments studying why this the case.
2.3.4 OUTPUT TYPE:
Since we know that output type can impact the transferability rate (ei et al. 2018 we hypothesize that a sequential output type (instead of single output label) can influence as well. This type of output is commonly found in ASRs. For an adversarial sample to transfer, the victim ASR needs to assign the attacker chosen character to each frame of the adversarial audio. This might increase the chances of a mistake, which for even a few frames, could lead to semantically wrong transcript (e.g., "occaaattit" "hhhaaatttt").
3

---

2.3.5 VOCABULARY AND SEQUENCE SIZES:
The last factor we consider is the output complexity or the total number of output labels. Transferability becomes harder as the number output labels increases (Demontis et al, 2019 Wu et al., 2018; Wu & Zhu 2020 Kurakin et al. 2016b). However, the output labels are not lixed in ASRs, as they are in the case of images (e.g., 10 output labels for the MNIST dataset). This is because ASRs are designed to output variable size sequences to account for the variation in the imput audio length. As a result, instead of considering total output labels, we use two metrics to capture output complexity: vocabulary and sequence sizes (Figure [e) and (f)). The vocabulary size is the number of unique words in the training data that the ASR has learned recognize. Similarly, the sequence size is the average number of words in each audio sample in the dataset. Both of these variables work in tandem to account for the ASR's output complexity.
STUDY DESIGN
Having outlined the six potential factors, we can now design an ablation study to measure their impact transferability. Designing this study is non-trivial since need to account for the 11 known factors from existing literature (Section 2.1 that limit transferability. If ignored, these factors alone can eliminate the transferability rade, hiding the effects of the aforementioned potential factors. We control for each of these 11 in our design.
Initially, we ran our ablation study on DeepSpeech, a real-world ASR which is commonly used in the adversarial research community. However, our experiments consistently yielded 0% transferability no matter what we did: removing or changing any of the ASR components did not change the transferability rate (Appendix A.2.). Upon further investigation, we realized this was primarily due to the large complexity of the model (a few million weights) and training data, which forced the transferability rate to remain unchanged 0%.
Overcoming this complexity was one of the major challenges of our work. We had to carefully design our experiments to remove the impact of complexity that comes with real- world ASRs, while simultaneously exposing the hidden factors limiting transferabiliit. As consequence, we had to run our study on simple yet realistic ASR desigus to uncover the factors impacting transferability.
Dataset: Since transferability becomes harder with the total number of output labels (De- montis et al. 2019 Wu et al.) 2018 Wu & Zhu, 2020 Kurakin et al.) 2016b), we use the small Google Speech Commands dataset (Warden 2018). This consists of clean, short audio files, cach containing one of 30 unique labels. Each audio file is one second long (or a vector of size ~16,000), either containing a number word. For the control experiment, we only choose a subset of the labels, specifically the numbers ONE to NINE (a total of nine labels).
Model Architecture: With this data, we can now train simple number recognition ASR We use the same architecture (Figure [ typically found in existing adversarial research papers (Abdullah al. 2021bbThe model outputs one of nine labels for a single audio.
Additionally, we account for the known factors that hinder transferability. To do so, we train a small model of approximately 250,000 trainable parameters and do not include regularization (e.g., dropout) to limit the effects of complexity and regularization. To reduce model complexity further, instead of using complex RNN cell types (c.g., LSTMs (Gers et al., 1999: and GRUs (Chung et al.] 2014t), we use the vanilla RNN cell (sim as RNN the remainder of the paper). We train five instances of the ASR on the exact same setup and hyper-paramesd
2021 (relerred to
(architecture, random seed, epochs, batch size, training data slice, etc). Each of these is trained to the standard real-world ASR accuracy.
For experiment, we can choose from several existing audio do- main optimization attacks. Since some of these attacks are architecture specific (Abdullah) et al.) 2020), they have the potential to bias results. As consequence, we formulate a generic optimization attack that captures the intuition of existing works. This helps extend our results to all existing audio domain optimization attacks.
Attack Formulation:
4

---

To do so, we first outline the steps used by optimization attacks, which generally follow the same approach. First, a perturbation is produced by minimizing:
| + + 6,
(1)
where is the loss function, the original audio sample, and is the target label. Next, 5 is clipped by magnitude factor a E R to control the quality of the adversarial audio:
ô = clip(,, a, -a)
(2)
Lastly, the is added to the original audio x and clipped to create a valid sample aadv!
Tadv clip(x + 1,1)
(3)
The clip step in Equation is used to control the quality of the adversarial audio and is often specific to the target model architecture. For example, attacks that clip spectral gradients (Qin et al., 2019) can not work against end-to-end ASRs that do not have spectrum generation slep (Abdullah et al. 2020). Therefore, in our generic attack, we making model agnostic.
remove Equation 2
Adversarial Audio Generation: We use this generic optimization attack to create adver-
sarial samples for each of the five ASRs. Following prior work (Abdullah et al. 2021b Liu et al. 2016). we only perturb audio samples that all of the ASRs transcribed correctly. We attack every label the dataset since some sample labels could be easier to perturb and some are casier to transfer (Carlini et al. 2019). For example, we perturb an audio sample containing the label ONE to produce each of the remaining labels TWO to NINE. Rumning the experiments in this exhaustive manner allows to generalize our findings.
We run the attack for 500 iterations. We save the adversarial sample every 50 iterations since the number of attack iterations can impact transferability (Dong et al. 2018). We also ensure that every saved adversarial sample has confidence greater than 0.99 because lower confidence adversarial samples are less likely to transfer (Abdullah et al. 2021b). In total, we create 4050 adversarial audio samples.
Having created the adversarial samples for each of the five ASRs, we transfer them to the remaining four models. Targeted transferability is successful if both the surrogate and the target ASRs output the same target label. However, adversarial samples exhibit asymmetry (Wu et al. 2018 Wu & Zhu, 2020), which is when an adversarial sample generated for model transters to model B, but not the other way around. To account for this, average the number of successfully transferred samples between all five models. This final average is the transferability rate.
Transferability:
4
EXPERIMENTAL ANALYSIS
Having described our design choices, we can now run the ablation study to observe the impact of each potential factor. This will involve changing or removing (as needed) each of the potential factors and recording the change in the transferability rate. It is important to note that this does not impact ASR accuracy because of two reasons. First, we retrain the model every single time. Second, the modified architecture resembles one used in current. literature. We will point to these papers each section.
4.1 CONTROL EXPERIMENT
Sctup: For the ablation study, we run control experiment to get bascline transferability rates. We use the same design choices (dataset, model architecture, attack formulation, adversarial audio generation, and transferability) from the previous section. This pipeline resembles the generic ASR pipeline attacked in current adversarial literature (Hannun et al. 2014]. We train and attack five instances of the control model and calculate the correspond- ing transferability rate. The ASRs achieve an average accuracy 95% and an agreement of 92%.
Results: Figure a(a) shows the transferability rate for our control experiment which is at 42%. Prior work has shown that transferability of optimization attacks is close to im- possible in real-world ASRs (approximately 0%) (Abdullah et al.) (021b).. We show that transferability is still possible for very simple ASRs too simple for the real-workd.
5

---

Transferability Rate (%)
99
100
88
80
60
60
40
19
19
20
(a) Control
(b) Input
(c) MFCC
(d) RNN Removed
(e) Output Type Changed
(f) Vocab Size Increased
(g) Sequence
Type Changed
Removed
Size Increased
Potential Factor
Figure 2: The target transfcrability rate of optimization attacks for every potential factor. We compare each factor's impact on the transferability rate against the (a) control experi- ment. There are three key insights. First, image domain samples (b) are harder to transfer
than audio ones. Second, removing (c) MFCC, (d) RNN, and (e) sequence output type
improves transferability. Third, increasing the (f) vocabulary and (g) sequence sizes reduces transferability.
Having measured the baseline results, we can now study the impact each of the potential factors (Section 2 We will only modify one potential factor at time while keeping the rest of the control setup unaltered. This will ensure that the change in the potential factor alone affects the transfcrability rate.
4.2 INPUT TYPE CHANGED
Here, we will switch from the audio dataset to an image one. We use the same setup the one used for the control experiment, except for the training data. Instead of training the ASR on audio samples containing numbers, we train it on images of handwritten numbers from the NIST Special Database (Grother 1995). This dataset is similar the MNIST (LeCun 1998 except that the size of the images is 128x128. To match the one- dimensional shape of audio (vectors size 16,000), we rasterize the images into single vectors of length h 16,384. We did not use the MNIST cataset because its sma smaller-sized images (28x28) rasterize to a length of only 784, which is substantially shorter than the audio samples. We train five instances of the ASR to mean accuracy of 93% and agreement of 92%.
Setup:
The results show that the input type does impact the transferability rate, with images being harder to transfer than audio samples. Specifically, the rate falls from 43% for audio (Figure 2 a))) to 19% for images (Figure bb) This fall can be attributed to the contrast. in the composition of audio and image samples. Specifically, image samples are composed of feature correlations that become periodic when we convert the images into one-dimensional vectors. Pixels that were next to each other in the original image are far apart in the vector. This can impact the MFCC extracted features and the corresponding robustness of the decision boundary.
Results:
4.3 MFCC REMOVED
Sctup: In this experiment, we will observe the MFCC's effect on transfcrability by removing from the pipeline. We use the same setup as the one used for the control experiment, except for the model architecture. We replace the MFCC with trainable layers, effectively converting the ASR into an end-to-end model seen in prior work (Amodei et al. 2016). It is important to note that this end-to-end model has greater complexity due to the inclusion of additional trainable layers. However, the MFCC too acts as a pre-trained, frozen layer since it contains handcrafted mel-filters MMannell 1994). In contrast, the trainable layer

---

allows learning filters during training. We train five instances of the is ASR to an average accuracy of 95%, and an agreement of 94%.
Results: The presence of the MFCC does hinder transferability (Figure 2). We
this as the rate increases from 42% (Figure (a)) to 60% (Figure 2(c)). Further experiments
in the Appendix demonstrated that this because the MFCC is regularizing the feature voctor, enables greater robustness to transforability.
4.4 RNN REMOVED
Sctup: We remove the RNN from the pipeline and observe the change in the transferability rate. We use the same setup as in the control experiment, except we modify the model architecture: replace the RNN with a convolutional layer to maintain the approximate number of trainable parameters. This converts the pipeline from a sequence-lo-sequence mapping type, to repeated one-to-one mapping type and resembles valid speech pipeline proposed in other works (Collobert et al. 2016). We train five instances of the ASR, to an average accuracy of 94%, and an agreement of 91%.
Results: Removing the RNN from the ASR improves the transferability rate from 42% (Figure 2 a)) 88% (Figure d)). This demonstrates that the RNN does limit transfer- ability. It is also important to note that we use vanilla RNN cell in the control, which we replace with CNN for this experiment. Furthermore, real-world ASRs use more complex RNN cells (e.g., GRUs and LSTMs) that include additional weights. Since model com- plexity hinders transferability (Wu et al. 2018 Demontis et al. 2019 we expect that the additional parameters of the GRU and LSTM cells will decrease transferability even further, enabling even greater robustness in real-world models.
4.5 OUTPUT TYPE CHANGED
To study the impact of the sequence output type, replace it with single-label one. Instead of outpulting a sequence of characters per frame, the pipeline will output single classification label (ONE to NINE) for the entire audio. We use the same setup as the one used for the control experiment, except we modify the final layer of the model architecture. The resulting ASR will produce single output for entire input, effectively converting the pipeline from a sequence-to-ssquees
Setup:
mapping to a one-to-one type. While this sctup may scem unusual, speech pipclines have used this approach in past works (Chen
et al. 2014 Higgins & Wohlford 1985 Rose & Paul 1990).
Results: The transferability increased from around 42% (Figure aaa) to almost 99% (Fig- ure 2 e)). This rate is almost identical to the 100% we observed in the image domain (Supplementary Materials Section A.3) and suggests that task type plays a very significant role in the transferability rate. A model that uses a sequence labeling task, instead of a single label one, will be more robust to transferability. Additionally, sequence labeling is necessary for ASRs due to the variability of speoch. Therefore, training an ASR as in the current experiment not possible for anything but trivial systems.
4.6 VOCABULARY SIZE INCREASED
Sctup: The output complexity can be measured using two variables, one of which is vo- cabulary size (Section 2.3.4). In this experiment, we observe the impact of vocabulary size by increasing its value. We use the same control setup, except that we increase the total number of output labels from 9 30. We do so by using the full Google Speech Commands dataset. We train five instances of the ASR, to an average accuracy of 92%, and agreement of 87%. This reduction in agreement from 92% (control experiment) to 87% is normal. By increasing the number of output labels, increase the likelihood of even the benign samples being labelod incorrectly.
Results: We compare the control (trained on 9 output labels) with this modified setup (trained on 30 output labels). We can observe that the transferability rate fell by 24 points, from 42% (Figure 2 a)) to 19% (Figure ()). Certainly, part of this drop be attributed
7

---

to the fall in agreement, from the control (92%) to the current setup (87%). However, a five percent drop in the agreement can not alone result in a large 24 percent drop in transferability. Therefore, a we believe that substantial degree of the drop is due to the increased vocabulary size. Furthermore, real-world data sets contain hundreds of thousands of unique words (Panayotov et al. 2015), far more than the 30 we tested in this experiment. Therefore, we can expect a further decrease in transferability in more realistic training data sets.
4.7 SEQUENCE SIZE INCREASED
Setup: In addition to vocabulary size, the sequence size also contributes towards output complexity. Similar to the previous experiment, we will study the impact on sequence size by increasing its value. We use the same setup as we had in the control experiment, except that we modify the original numbers-only training dataset. We generate a new dataset containing sequences of four numbers by concatenating audio samples (and their corresponding labels). For example, we concatenate four andio files that contain the words ONE, NINE, ONE and NINE, resulting in single audio file containing the sequence ONE NINE ONE NINE. We train five instances of the ASR to an average accuracy of 95%, and agreement of 86%.
Results: Increasing the size of the sequence by just single word can reduce the transfer- ability rates from 42% (Figure 2 a)) to a mere 4% (Figure g(g) Real-world ASRs process much longer sequences than just size four. The average length of sequence in the English language is around 15 words (Cutts 2020). Therefore, due to the increased sequence size, can expect an even further decrease in the transfcrability rate in real-world ASRs.
DISCUSSION AND TAKEAWAYS
In the previous section, we explore how the six factors impact the transferability of opti- mization attacks in ASRs. In this section, we will discuss the takeaways from our findings.
An ablation study to explore factors impacting transferability must be per- formed on a simple, but realistic ASR pipcline. Real-world ASRs are large and complex, which itself limits the transfcrability of optimization attacks to an abysmal 0%. Performing an ablation study on such models will yield no results since removing or chang- ing any component will not change the 0% transferability rate. This will hide the impact of other unknown factors that might be playing a role. As a result, we run an ablat ion study on simple ASR pipeline and uncover six previously unknown factors that limit transferability.
Existing optimization attacks are unlikely to provide targeted transferability against real-world ASRs. In this paper, uncover a number of factors that limit the transferability of optimization attacks in ASRs. Interostingly, these same factors are known to improve accuracy, and therefore, will likely be found in most real-world ASRs (Hannun et al. 2014). As a result, real-world ASRs will likely remain robust to oplimization allacks, motivating the need to our attention to other altack lypes.
The factors preventing transferability are also required for the correct function- ing of the ASR. Output type, large vocabulary, and sequence sizes are crucial for training accurate general-purpose ASRs. This is because ASRs need to account for varying audio length (output type) and variety of words (vocabulary size) and phrases (sequence sizes). It might not be possible to train any wollworld ASR without these three components. For- tunately, these same components also prevent transferability. Therefore, real-world ASRs will be robust to the targeled transferability of existing optimization altacks.
Sceking better attacks: Since optimization attacks do not provide targeted transferabil- ity in ASRs, the community should focus on the attack types that can. Signal processing attacks (Abdullah et al. 2021a 2019) is a family of attacks unique to the audio domain. These attacks exploit the feature extraction layer of the ASR pipeline (Figure [b) and therefore, provide key advantages. These attacks not only provide targeted transferability, but also require fewer queries, are model agnostic, require only black-box knowledge, and take mere seconds to execute (Abdullah et al.) 2021b). However, clean, targeted signal processing attacks (i.e., adversarial audio that sounds clean to humans and transcribed
8

---

as the targeted text) do not yet exist. Therefore, developing such attacks (to replace opti- mization ones) is a potential direction for future research.
Building speaker recognition models that are robust to targeted transferability of optimization attacks: Speaker recognition is one application in the audio domain that has very similar pipeline to that of ASRs. However, unlike ASRs, speaker rocognition models (specifically text-independent oncs) are not robust to targeted transfcrability of op- timization attacks (Chen et al., 2019). Text-independent models check for speaker identity, without verifying the text in the audio sample. As a result, these models do not employ sequence labeling, vocabulary, and sequence sizes, the absence of which enables transferabil- ity. As result, it is possible to build robust speaker recognition models by simply including text. verification as part of the pipeline. This will involve using an ASR for text verifica- tion, which can then be followed by speaker identification. Since target transferability will fail at the ASR level, the speaker identification will not be triggered. This will ensure the robustness of the overall speaker recognition pipeline.
RELATED WORK
ASRs are vulnerable to adversarial samples that can force them to output malicious labels. In the audio domain, there three types of attacks to generate such adversarial samples:
signal processing attacks (Abdullah et al. 2021a 2019), gradient-free attacks TTooi et al.
2018
Alzantot et al. 2017 Chen al. 2020 and optimization attacks (Carlini & Wagner
2018
Cissé et al.) 2017
Kreuk et al.] 20 Sa Qin et al. 2019 Schönherr et al. 2019 Abdoli
et al.
2019
Yuan al. 2018 Yakura Sakuna
2018 Alzantot et al.) 2017
Signal Processing Attacks: do exhibit largeted transferability because these exploit the feature extraction stage of the ASR pipeline. They produce adversarial samples whose feature vectors are similar to the ones produced for the benign sample. As consequence,
the model unable to ascertain whether the feature vector came from benign or adversarial audio. However, there do not yet exist signal processing attacks that can produce clean audio that is also transcribed as the adversary chosen text. So far, existing signal processing attacks can only generate targeted noisy audio (i.e., the attack audio sounds like noise to the human ear, but transcribed as the targeted text by the model) or untargeted clean audio (i.c., the attack audio sounds clean to the human car, but is transcribed as garbage text by the model).
Gradient-Free Attacks: craft adversarial samples by repeatedly querying the target model. Theoretically, this can enable attackers to exploit models in black-box settings. However, these attacks have not had any success against real-world ASRs. Additionally, these attacks have not demonstrated any transferability, even against the simple ASRs.
Optimization Attacks: use the model gradients to craßt adversarial samples. These at- lacks have been incredibly successful in the image domain (Szegedy el al., 2013 Goodfellow ltt al.] 2014 2016 Papernot et al., 2017 2016). One of the reasons for this is the ability of their samples to exhibit transterability even for real-world black-box systems [Lu et al. 2016). Most of the work in the space of transferable optimization attack samples has there- fore been focused on image recognition models. There are several factors that have been shown affect transfcrability, which we list Scction [2.1] In contrast, our work specifically focuses on the audio domain. We are the first to study the factors that make the targeted transferability of optimization attacks difficult against ASRs.
7 CONCLUSIONS
In this work, we investigated why ASRs robust to target transferability of optimization attacks. We conducted an exhaustive ablation study and uncovered previously unknown factors that limit transferability. The ability of these factors to limit transferability exposes scrious limitation in optimization attacks against ASRs. As result, attack types that have demonstrated transferability in real-world settings, like signal processing attacks, deserve more attention.
9

---

8 ACKNOWLEDGMENTS
This work is partly supported by the National Science Foundation under CNS- ANS-1933208. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. Additionally, would like thank Patrick Emami for providing valuable comments and revisions on an early draft of this work.
CODE OF ETHICS
These authors have reviewed and adhered to the ICLR Code Ethics.
10 REPRODUCIBILITY STATEMENT
To train the ASR, the readers refer to the pipeline found in the Supplementary materials, Figure 3. This figure describes the exact ASR architecture we used in the experiments. The information about the training data, ASR pipeline, attack formulation, and attack hyper parameter details cxist in Section 3.
REFERENCES
Simplernn, 2021. Available at keras/layers http//yyyyyrrr///eataa.. SimpleRNN
Sajjad Abdoli, Luiz G Hafemann, Jerome Rony, Ismail Ben Ayed, Patrick Cardinal, and Alessandro L Koerich. Universal Adversarial Audio Perturbations, arXiv preprint
arXiv:1908 19110119080 2019.
Hadi Abdullah, Washington Garcia, Christian Peeters, Patrick Traynor, Kevin Butler, and Joseph Wilson. Practical Hidden Voice Attacks against Speech and Speaker Recognition Systems. Proceedings of the 2019 Network and Distributed System Security Symposium (NDSS), 2019.
Hadi Abdullah, Muhammad Sajidur Rahman, Christian Peeters, Cassidy Gibson, Vincent Bindschaedler, Washington Garcia, Tom Shrimpton, and Patrick Traynor. Beyond Ip clipping: Equalization- based psychoacoustic attacks against asrs. 2020.
Hadi Abdullah, Muhammad Sajidur Rahman, Washington Garcia, Logan Blue, Kevin War- ren, Anurag Swarnim Yadav, Tom Shrimpton, and Patrick Traynor. Hear "no evil', see "Kenansville': Efficient and transferable black-box atlacks on speech recognition and voice identification systems. IEEE Securily and Privacy (SBP), 2021a.
Hadi Abdullah, Kevin Warren, Vincent Bindschaedler, Nicolas Papernot, and Patrick Traynor. SoK: The Faults in our ASRs: An overview of attacks against automatic speech recognition and speaker identification systems. IEEE Security and Privecy (SBP), 2021b.
Moustafa Alzantot, Bharathan Balaji, and Mani B. Srivastava. Did you hear that? Ad- versarial Examples Against Automatic Speech Recognition. In Nearal Information Pro- cessing Systems Workshop on Machine Deception 2017, volume abs 11011800005544 Neural Information Processing Systems, 2017. URL http://arxiv. /tt///aawww..rr/a////.....00.. 00554
Amazon. Amazon Alexa, 2021. Available https://wwwt wwwwon.aaaa..ommmmmmmm......... wtt.://oommoooommmmmmomom
21841306011
Dario Amodei, Sundaram Ananthanarayanan Rishita Anubhai, Jingliang Bai, Eric Batten- berg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, el al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International Conference on Machine Learning, pp. 173-182, 2016.
Raghav Bharadwaj. Voice and Speech Recognition in Banking What's Possible To- day, 2019. Available at https: /teps://emerrj com/ http:/aaassetrreeeeeeeeeeeeerree wtwpooooovrviiisooooooo..o speech-
iecognitionnbanniin
10

---

Tom B Brown, Dandelion Mané, Aurko Roy, Martin Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint
2017.
Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks speech-to-text. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 1-7. IEEE, 2018.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dim- itris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint Xie: 1 000000000... 2019.
Guangke Chen, Sen Chen, Lingling Fan, Xiaoning Du, Zhe Zhao, Fu Song, and Yang Liu. Who is real bob? adversarial attacks on speaker pecognition systems. arXiv preprint 1rrXi:1911..11 000040, 2019.
Guoguo Chen, Carolina Parada, and Georg Heigold. Small-foolprint keyword spotting using deep neural networks. In 2014 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4087-4091. IEEE, 2014.
Yuxuan Chen, Xuejing Yuan, Jiangshan Zhang, Yue Zhao, Shengzhi Zhang, Kai Chen, and XiaoFeng Wang. Devil's whisper: A general approach for physical adversarial at- tacks against commercial black-box speech recognition devices. In 29th USENIX Security Symposium (USENIX Security 20), 2020.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Moustapha Cissé, Yossi Adi, Natalia Neverova, and Joseph Keshet. Houdini: Fooling Deep Structured Visual and Speech Recognition Models with Adversarial Examples. In Ad- vances in Neural Information Processing Systems 30: Anrvual Conference on Neural In- formation Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6980 6990, 2017.
Ronan Collobert, Christian Puhrsch, and Gabriel Synuaeve. Wav2letter: an end-to-end couveet-based speech recognition system. arXie preprint arXiv: 031993, 2016.
Martin Cutts. Oxford guide to plain English, Oxford University Press, USA, 2020.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina. Nita-Rotaru, and Fabio Roli. Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks. In 28th {USENIX) Security Symposium (( USENIX) Security 19), pp. 321-335, 338,, 2019.
Yimpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1185-9193, 2018.
Dan Froomkin. The Computers are Listening, Part 2015. Available at https: /ttte//trrrar..cca/////6//////aa
caw.rrrrrarrppppi--ameeeeaaaaaa american-phone-clll
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. 1999.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXie preprint orXiv:1412.6572, 2014.
Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick Mc- Daniel. Adversarial examples for malware detection. In European symposium on research in computer securily, pp. 62-79. Springer, 2017.
11

---

Patrick J Grother. Nist special database 19. Handprinted forms and characters datobase, National Institute of Standords and Technology, pp. 10, 1995.
Chuan Guo, Jared S Frank, and Kilian Q Weinberger. Low froquency adversarial perturba- lion. arXiv preprinl arXiv: 1809. 180.00858, 2018.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint 142.55677, 2014.
Alan Higgins and R Wohlford. Keyword recognition using template concatenation. In ICASSP 85. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 10, pp. 1233 1236. IEEE, 1985.
Weiwei Hu and Ying Tan. Generating adversarial malware examples for black-box attacks based on gan. arXie preprint orXiv: 102::700.05099 05983, 2017.
Felix Kreuk, Yossi Adi, Moustapha Cisse, and Joseph Keshet. Fooling End-to-end Speaker Verification by Adversarial Examples. arXiv preprint ar 1801 181.0090,,,,39, 2018a.
Felix Kreuk, Assi Barak, Shir Aviv-Reuven, Moran Baruch, Benny Pinkas, and Joseph Keshet. Deceiving end-to-end deep learning malware detectors using adversarial examples. arXiv preprint arxiv:1802. 04528, 2018b.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint (tt::660700533 2016a.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint 161i:1611.01220 2016b.
Yann LeCun. The mnist database of handwritten digits. htlp://yann. lecun. /en/s//mm/mss/.....mmmmm 1998.
Yiqing Lin and Waleed H Abdulla. Principles of Psychoacoustics. In Audio Walermark, pp- 15 49. Springer, 2015.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint orXiv:161.002700 2016.
R.H. Mannell. The perceptual and auditory implications of parametric scaling in synthetic speech. Macquarie Universily, pp. (Chapter 2), 1994. URL "http://clas. wt..m...ee.. edu. au/ sppech/acousticss hetpgpoor_.eeeeetttttooottccccie
html"
Muhammad Muzammal Nascer, Salman H Khan, Muhammad Haris Khan, Fahad Shah- baz Khan, and Fatih Porikli. Cross-domain transferability adversarial perturbations. Advances in Neural Information Processing Systems, 32:12905-12915 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an
ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206 5210. IEEE, 2015.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transfcrability in Machine Learn- ing: from Phenomena Black-Bos Box Attacks using Adversarial Samples. orXiv preprint arXiv: 1605. 1ee1.00057..77 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical Black-box Atlacks Against Machine Learning. In Proceed- ings of the 2017 ACM on Asia Conference on Compuler and Comsnunications Securily, pp. 506-519 ACM, 2017.
Yao Qin, Nicholas Carlini, Ian Goodfellow, Garrison Cottrell, and Colin Raffel. Imper- ceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition. ar preprint arxi: 10346, 2019.
12

---

Richard C Rose and Douglas B Paul. A hidden markov model based keyword recognition system. In International Conference on Acoustics, Speech, and Signal Processing, pp. 129-132. IEEE, 1990.
Lea Schönherr, Katharina Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa. Ad- vensarial Attacks Against Automatic Speech Recoguition Systems via Psychoacoustic Hid- ing. The Internet Society, 2019. URL httpp::/www, nnss- synposium wtw.a/.rrggddddd/////
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. Fawkes: Protecting personal privacy against unauthorized deep learning models. arXiv preprint urxiv:2002. Xiv:2002.08327, 2020.
Yash Sharma, Gavin Weiguang Ding, and Marcus Brubaker. On the effectiveness of low frequency perturbations. arXiv preprint arXiv:1933000 000000 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.66 2.999. 2013.
Rohan Taori, Amog Kamselty, Brenton Chu, and Nikita Vemuri. Targeted Adversarial Examples for Black Box Audio Systems, arXiv preprint ar 07820, 2018.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprinl 1704:1704.03450 2017.
Aliza Vigderman. Best Alexa Home Security Systems, 2021. Available at httpp://www security.
Pote Warden. Speech commands: datasct for limited-vocabulary speoch recognition. Xr preprint wrXiv:1804.03200 2018.
Xingxing Wei, Siyuan Liang, Ning Chen, and Xiaochun Cao. Transferable adversarial at- 2018.
tacks for image and video object detection. arXiv preprint
Lei Wu and Zhanxing Zhu. Towards understanding and improving the transferability of adversarial examples in deep neural networks. In Asian Conference on Machine Learning, PMLR, 2020.
pp.
Lei Wu, Zhanxing Zhu, Cheng Tai, et al. Understanding and Enhancing the Transferability of Adversarial Examples. arXiv preprinl 1xi:::802200910 2018
Hiromu Yakura and Jun Sakuma. Robust Audio Adversarial Example for a Physical Attack. arXiv preprint 1810v:1800.1 1188111111111 11793, 2018.
Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, and Carl A Gunter. CommanderSong: A System- atic Approach for Practical Adversarial Voice Recognition. In Proceedings of the USENIX Security Symposium, 2018.
Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In Proceedings of the European Conference Computer Vision (ECCV), pp. 452-467, 2018
A SUPPLEMENTARY MATERIALS
A.1 RNN AS A POTENTIAL FACTOR
A typical ASR neural network is made up of the following four stacked compo- nents:CNN- Sants:OMN-XRN--PFC--20 >RNN-SF-XLLOIISS as scen in Figure iii.. It is not necessary to exhaustively study the impact the transferability of each of these components. This is because transfer- ability in image models is far easier (Supplementary Materials A.1) than in ASRs Abdullah et al. (2021b). This suggests that. there component(s) unique to ASRs (i.e., components
13

---

non-existent in image models) that might be hindering transferability. As consequence, we focus our study on the symmetric difference of the two model architectures (i.e., present in ASRs but not in image models). The only component that fits this criterion is the RNN, as shown in Figure [c).
A.2 REAL WORLD ASR FOR ABLATION STUDY
In the ideal case, we would want to run the ablation study on real-world ASRs, since are trying to make conclusions about such ASRs. However, such study will provide inconclu- sive results about the potential facotors. A real-world ASR has high model complexity, with tens of millions of training parameters. Similarly, it is trained on complex data sets (i.e., containing a large vocabulary size of hundreds of unique words and long sequence sizes). We already know that large model and output complexity limit transferability to 0% Abdullah et al. (2021b). As a consequence, if we were to run an ablation study on such an ASR, the model and output complexity alone will force the transferability to remain at 0%. For example, while removing the RNN increases transferability (Figure 4 this step will have no impact transferability in the case of real-world ASR.
To validate this hypothesis, we conducted a simple experiment. We trained multiple real- world ASRs (on identical setups) that did not contain an RNN using the wav2letter chitecture Collobert et al. (2016)- We used the LibriSpecch data set for training, which contains 1000 hours of audio, has vocabulary size of 900,000 unique, and up sequence size of 20. We followed the same methodological steps outlined in Section 3 to produce the adversarial samples. Of these, none of them transferred from the surrogate the target models we had trained.
As a consequence, we had no choice but run the study this phenomenon on a simpler ASR using a less complex training data set. This limits the impact of the known factors and helps isolate the impact of the unknown ones.
A.3 TARGETED TRANSFERABILITY FOR TMAGE MODELS
We know that targeted transferability rates for audio models are abysmally low Abdullah et al. (2021b), even when the models are trained on identical setups. We conduct the tollowing experiment to study whether this is true for image recognition models:
A.3.1 SETUP
We use the same general setup we had used in the control experiment (Sec- tion 4.1), except change the architecture, the training data, and the at- tack. We train on the MNIST LatasetLeCun (1998) on the following architec- SAA-PPODD-CCNNMMMXXPP-O---------
The model contains 250,000 trainable parameters, similar to the control. We train the models for 12 epochs, with 1024 batch size an accuracy of 94% and an agreement 89%. We use the basic iterative attack Goodfellow et al. (2014) clipping perturbations at values of 0.1 and 0.05.
ture:
A.3.2 RESULT
We observe a transferability of exactly 100% for both clip values, for all iterations. In stark contrast, audio transferability is close to 43% (Section 4.1). This despite the fact that the attack we use against the image models uses clipping. In contrast, we perform no such clipping in the audio allack (Section 3.4). To add to that, the accuracy and agreement of the image model (94% and 89%) is lower than the control (95% and 92%). This implies that there are additional previously unknown factors that are preventing transferability in the audio domain.
A.4 MFCC AND TRANSFERABILITY FOR ASRs
transferability rate remains constant for the control, when the MFCC is present (Fig- ure B a)). However, when the MFCC is removed, the transferability rate drops from 65% (at.
14

---

LOGITS
out size 29
FC kernel width 32 filters
RNN numbez of filters 50
CONV
kernel width 7
stride length 1 number of filters 50
CONV kernel width stride length 1 number filters 50
CONV kernel width 7 length number of filters 50
CONV stride length 2 filters 50
MFCC
Figure 3: The details of the pipeline we used in the control experiment. The pipelines for the potential factors we study in the paper are all based the one shown above.
50 iterations) to 57% (at 500 ilerations), shown in Figure 5 b). This is because increasing iterations produce "over-fitted" adversarial samples. This suggests that the MFCC is regu- larizing the feature vector, whereby helping the model learn smoother decision boundaries robust to target transferability.
15

---

Results for Potential Factors
100
(e)
(d)
Transferability Rate (%)
80
(c)
60
40 (a)
20
(8)
(9)
50
100
150
200
250
300
450
400
450
50
Iterations
(a) Control Experiment
(b) Domain Type Changed
(c) MFCC Removed
(d) RNN Removed
(e) Sequence Labeling Task Removed
(f) Vocabulary Size Increased
(g) Sequence Size Increased
Figure 4: We can observe the change in the transferability rate with respect to the number of iterations. Generally, we see that the number of iterations does not have a significant impact on the transferability rate, except when the MFCC is removed.
16

---

Results for Potential Factors
(a) Control Experiment
100
(b) MFCC Removed
Transferability Rate (%)
80
(b)
60
40 (a)
20
0
50
100
150
200
250
300
450
400
450
500
Iterations
Figure 5: The plot shows the relationship between transferability and iterations for two tups: the control experiment and when the MFCC is removed. For the control experiment the transferability rate is consistent across all iterations removed, the rate falls as the iterations increase (b). This suggests that the MFCC is regu- larizing the feature vector helping the model learn smooth and robust decision boundaries.
(a). However, when the MFCC is
17


Reviews:
[{"review_id": "tpsqAUomS1O", "reviewer": "Reviewer_Vmod", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an important problem and provides valuable insights into the mysterious fact of the very low target transferability between ASR models. The control experiments are carefully executed and provide clear takeaway insights. The paper is also well-written and nicely structured. Below are my comments that can further improve this paper:\n1. The experiments on the input type do not seem very interesting to me. Images are inherently 2-dimensional. Converting them to 1-dimensional naively can destroy the locality. There are more advanced ways to convert and an image to 1D, such as using a Hilbert curve.\n2. The paper only uses the vanilla attack. There have been quite a few papers in the adversarial machine learning literature on improving transferability (e.g., Expectation over Transformation, attacking an ensemble, etc). \n3. This paper only uses a single model, DeepSpeech, which is a character-based model. A word-based model may have different behaviors.\n", "summary_of_the_paper": "This paper studies why optimization-based adversarial attacks have close to zero target transferability in attack real-world ASR pipelines.\nThe paper identifies 6 previously unknown factors that impact target transferability, including the input type, Mel Frequency Cepstral Coefficient (MFCC), the Recurrent Neural Network (RNN), output type, and the vocabulary and sequence sizes. \n", "summary_of_the_review": "A good paper that studies an important problem with carefully executed control experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Es3zaROu0Ss", "reviewer": "Reviewer_qpFd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors design a simplified ASR system in order to show which components of\nsuch a system are affecting the targeted transferability issue. First, a\nsimplified ASR system is required, as it has yet to be shown that complex\nreal-world ASR systems have 0% targeted transferability.\n\nThe writing is clear, and the design is simple. The results provide some evidence\nas to which components contribute to targeted transferability. \n\nHowever, some of the choices in the modeling and explanations are a little weak in \nsome areas:\n\n1. The model is not fully specified within the text, only mentioned in a reference.\nThe model should be at least briefly summarized. In the least explain how the model\ngoes from frame level output labels to a label (CTC? RNN-T?).\n\n2. The authors discuss vocabulary size, but is this even handled within their model?\nThe model to me appears to only have 9 labels which are synonymous with vocabulary size.\nAlso, since there are no details of the model within the paper, how is the alignment issue handled?\n\n3. What about attacks on the mfcc themselves?\n\n4. Why wasn't there an ablation study on more complex RNN cells (LSTM, GRU). Should have been straightforward \nto run this test.\n\n5. Why is there no ablation study on the effects of dropout?\n\nSmaller points:\n\na. In the description of input types, what is the significance of having topographical input with respect\nto adversarial attacks.\n\nb. In 2.3.4, \" the victim ASR needs to assign the attacker chosen character to each frame of the adversarial audio\", \nbut this is highly dependent on the ASR model. For example in CTC or RNN-T, what excludes the possibility of\nmodifying only a handful of frames in order to affect the output.\n\nc. In 2.3.5 the frame level output for your simplified ASR model is indeed fixed. The output sequence is also\nfixed (albeit large).\n\nd. No reference for Deepspeech.\n", "summary_of_the_paper": "It is known that targeted transferability is ineffective against typical ASR\nsystems, but it is not fully understand why.  In this work the authors show the\nimpact of targeted transferability against a simplified ASR system.\nSpecifically, they perform an ablation study to assess which components of an\nASR may be contributing to the non targeted transferability issue.  Their\nresults provide insights into ASR case, which may help against other\ntransferable attack types, and also help in other domains.\n", "summary_of_the_review": "I believe the authors provide a useful beginning of understanding why ASR systems may be \nnot susceptible to targeted transferability. However, I think a little further work\ncould have been done to improve the results as well as the explanations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A4-n8SzNnuB", "reviewer": "Reviewer_9PZL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+ The paper is well-written and easy to follow.\n\n+ The paper gives some takeaways about the limited adversarial transferability in ASR systems.\n\n+ The paper does lots of work for the system implementation and testing.\n\n+ The paper gives many details in implementation and evaluation.\n\nWeaknesses & comments:\n- I basically like the ideas and findings presented in this paper. However, I think this paper focuses on the measurement of ASR systems and may be more suitable for a security conference. \n\n- The method of how the authors come up with the factors is not explained. This is the main limitation of the paper. \n\n- The definition of a “factor” is not clear and is not clearly defined in this paper. For example, RNN is a factor. Also, Output Type is a factor, but specifically, Sequential Output is a factor/reason? Same with Input Type, and Output Complexity. Is Vocabulary a factor, or is Complexity a factor? Isn’t the total number of Output Labels factor the same as Output Type factor? So please define what is a factor. Please also see my comment on EFA.\n\n- As a consequence of the above point, don’t you think “model complexity” is also a factor? Maybe even experiment with transferability Vs. increasing model complexity?\n\n- How does “Number of models” simultaneously optimized in Equation 1 affect transferability? Do you think this could improve transferability? (e.g. optimize jointly on 3 models, test on 4th model).\n\n- The ASR pipeline in Section 2.2 should be generalized. For example, call the RNN as “ASR model”, MFCC as “Signal Processing Algorithm” and so on. Also, how did the authors give this particular pipeline as an example? Is it something that they have observed in existing literature?\n\n- I would like to see an EFA analysis on these factors. What are the groupings of these factors?\n\n- The second factor, MFCC, should be generalized. I am not an expert in this field, but I suppose there are more algorithms than just MFCC that do this operation. \n\n- I don’t understand the RNN factor, which is not clear. Is the RNN itself a factor, or any sequence model, such as RNN, LSTM … a factor?\n\n- The current evaluation is not extensive. The paper can benefit from giving a deeper evaluation of these identified factors. For example, what are the detailed impacts of each known factor in the control experiment? What will happen if you change two factors at the same time, e.g., remove MFCC but use a complex RNN? And what is the impact of using RNN with different complexity?\n", "summary_of_the_paper": "This paper presents an evaluation and explanation of the limited adversarial transferability in the ASR system. It first lists 11 known factors, e.g., smoothness of gradients, and then proposes four potential factors that limit the transferability. Then it examines the transferability in the DeepSpeech system and found its model is too complex for adversarial transferability. Hence, the author implements 5 ASR systems based on a simple model trained from the Google Speech Commands dataset. It generates 500 adversarial audio samples for each of the five ASRs and tries to transfer samples to others. The evaluation examines the target transferability rate of optimization attacks by one factor at a time. Then it gives some insights into the impact of each of the potential factors.", "summary_of_the_review": "Although this paper looks simple, it has actually identified an interesting, novel problem. I think the authors motivated this paper from a recent SOK paper published at the IEEE S&P'21 conference that has discussed related issues. I appreciate that the authors have done research on such an important topic. I would suggest accepting this paper as a short paper or a poster if possible.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-fFK0zWVdl5", "reviewer": "Reviewer_TNL7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- A timely study on an important problem has been highlighted recently.\n- Ablation evaluation methodologies are systematic.\n- The finding on robustness improvement and transferability is interesting.\n\n# Weaknesses\n- Lack of in-depth root causes analysis on the findings. The paper provides some discussion in each ablation study. However, the current discussion does not go beyond describing the observations of the experimental results. For measurement papers, the readers are more interested (and will be more appreciated) if an in-depth analysis on the fundamental reasons why such feature affects the transferability. This is not an easy task, where additional small-scale experiments may also be required as the authors generate hypotheses.\n- The current discussion of the findings is limited to the ASR domain, however, as ASR and image tasks were discovered to have distinctive attack transferability properties, the author may consider drawing connections with the image space. For example, will the similar features also make transferability more difficult in the image tasks? Such analysis also helps to understand the unique challenges in the ASR and perhaps improve the robustness of image task models.", "summary_of_the_paper": "The paper conducts a systematic study on the phenomenon that attacks targeting ASR systems often have low transferability. To do that, the authors take a representative ASR pipeline and perform ablation studies by modifying or removing the components that may have an effect on the attack transferability. Results show that many existing designs for improving the robustness of ASR can also prevent transfer attacks. Based on the findings, the authors also discuss the takeaways and future directions for the ASR.", "summary_of_the_review": "The paper studies an important problem and presents many interesting findings. However, it still lacks root cause analysis on the findings to indeed “demystify” the low attack transferability in ASR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores why adversarial examples do not transfer well in
adversarial examples on automatic speech recognition systems. The authors
propose a number of potential causes that are then quickly evaluated in
turn.

This could be an excellent paper, but in its current form, it is borderline.
The main problem with the paper is that it proposes a number of causes for the
limited transferability, and then evaluates each of them with one quick
experiment and just a paragraph of text. In particular, none of the results
actually convince me that the claim is definitely correct, and many of the
experimental setups are confusing or would have other explanations other than
the one variable that is aiming to be controlled for.

That said, even with these weaknesses, this paper raises interesting and new
questions with an approach I have not seen previosuly. So while I don't believe
the paper has done much to actually demystify transferability, it does take
steps towards performing scientific experiments to understand why it is so
hard. And these experiments, while not perfect, can serve as the basis for
future work to extend and understand which factors are most important.
Decision: Accept (Poster)