Demystifying Limited Adversarial Transferability in Automatic Speech Recognition Systems

1 Introduction . It is hard to understate the pervasiveness of ASRs in security-critical systems . These include banking ( Amazon , 2021 ) , surveillance ( Froomkin , 2015 ) , online retail ( Vigderman , 2021 ) , and home security ( Bharadwaj , 2019 ) . However , ASRs are vulnerable to specially crafted adversarial samples , that force them to produce malicious outputs . In the research community , the most popular class of algorithms for this purpose are the optimization ( or gradient-based ) attacks ( Abdullah et al. , 2021b ) . This is largely because these attacks exhibit targeted transferability in some domains . Specifically , samples crafted for a local model ( surrogate ) can force a different black-box model ( target ) to produce the attacker chosen output . This is the case even if the surrogate and target use different architectures , training data , hyper-parameters , etc . Therefore , the transferability property has enabled attackers to exploit security-critical applications which include facial recognition systems ( Shan et al. , 2020 ) , image APIs ( Brown et al. , 2017 ; Liu et al. , 2016 ) , authentication systems ( Chen et al. , 2019 ) , and malware detectors ( Hu & Tan , 2017 ; Kreuk et al. , 2018b ; Grosse et al. , 2017 ) . In short , target transferability makes optimization attacks effective against several real-world systems . However , recent work has uncovered that these attacks do not exhibit target transferability between ASRs , even when the ASRs are trained on identical setups ( i.e. , same hyperparameters , architecture , random seed , training data ) ( Abdullah et al. , 2021b ) . Therefore , these attacks can not be used against black-box ASRs , casting doubt on the usefulness of the entire class of optimization attacks in the audio domain . However , the reasons for this failure is not understood . In this work , we study this phenomenon to uncover the factors that prevent the transferability of optimization attacks between ASRs . To ensure we can uncover each one of these factors , we perform an exhaustive ablation study on the entire ASR pipeline and observe the impact of the different components on the transferability rate . We test thousands of adversarial samples across multiple models and characterize six factors that impact the transferability rate . In doing so , we make the following contributions : 1 . We identify six previously unknown factors that impact target transferability . These include the input type , Mel Frequency Cepstral Coefficient ( MFCC ) , the Recurrent Neural Network ( RNN ) , output type , and the vocabulary and sequence sizes . These factors explain the near 0 % transferability rate seen in prior works . 2 . We highlight the relationship between accuracy and adversarial robustness in ASRs . The five factors that improve ASR robustness also improve accuracy . 3 . Our findings explain why one of the most popular classes of attacks , across the adversarial machine learning space , fails in the audio domain . These can be leveraged to strengthen models from other domains ( e.g. , images ) , that have traditionally been vulnerable to optimization attacks . We begin our study by listing all the factors from the existing literature that are known to hinder transferability ( Section 2.1 ) . Even when controlling for these factors , transferability rates in ASRs still do not achieve the near 100 % observed in the image models ( Section 4.1 ) . This suggests the existence of additional factors limiting transferability . We list the additional potential factors ( Section 2.3 ) , describe our design choices ( Section 3 ) , and through a series of ablation experiments , we systematically quantify the impact of each factor on transferability ( Section 4 ) . Based on our findings , we discuss several takeaways ( Section 5 ) , present related work ( Section 6 ) , and summarize our findings ( Section 7 ) . 2 Factors . 2.1 Known Factors From Existing Literature . Before delving into ASRs , it is first important to review the existing literature on transferability . While doing so , we identified 11 factors that are already known to limit transferability of optimization attacks : 1 . Smoothness of gradients ( Demontis et al. , 2019 ; Zhou et al. , 2018 ; Wu et al. , 2018 ) 2 . Attack type ( Kurakin et al. , 2016a ; Dong et al. , 2018 ; Liu et al. , 2016 ) 3 . Number of attack iterations ( Dong et al. , 2018 ) 4 . Number of output labels ( Liu et al. , 2016 ) 5 . Spectral makeup of the perturbations ( Sharma et al. , 2019 ; Guo et al. , 2018 ) 6 . Model architecture ( Wu et al. , 2018 ) 7 . Model accuracy ( Wu et al. , 2018 ) 8 . Model complexity ( Demontis et al. , 2019 ; Wu et al. , 2018 ; Wu & Zhu , 2020 ) 9 . Model agreement ( Tramèr et al. , 2017 ) 10 . Confidence of the adversarial sample ( Abdullah et al. , 2021b ) 11 . Asymmetry ( Wu et al. , 2018 ; Wu & Zhu , 2020 ) Unfortunately , outside of a single example ( Abdullah et al. , 2021b ) , the primary focus of most of these works has been image classification models . As a consequence , even when controlling for these factors , we observed that transferability between ASRs remains low ( Section 4.1 ) . This suggests the existence of unknown factors limiting transferability . 2.2 ASR Pipeline . To identify these unknown factors , we first provide a brief overview of the different components that make up the ASR pipeline ( Figure 1 ) . To that end , we consider the most commonly attacked ASR pipeline in the research community ( Abdullah et al. , 2021b ) . The first stage of an ASR splits the input audio into overlapping frames ( Figure 1 ( i ) ) . Next , a signal processing algorithm , ( e.g. , the MFCC ( Lin & Abdulla , 2015 ) ) extracts a feature vector from each of the overlapping frames ( Figure 1 ( ii ) ) . Next , the neural network ( Figure 1 ( iii ) ) assigns a single character label to each feature vector , resulting in a character list ( e.g. , “ hheellllo bbbboooob ” ) ( Figure 1 ( iv ) ) . These are then aggregated into a single word ( e.g. , “ hello ” ) , which is then combined with other words into a final sequence ( e.g . “ hello bob ” ) ( Figure 1 ( v ) ) . 2.3 Additional Potential Factors . Having described the ASR pipeline , we can now list the components that might potentially impact transferability . We will later experimentally investigate their impact during ablation study . These potential factors include : 2.3.1 Input Type : . Given previous findings ( Naseer et al. , 2019 ) , we hypothesize that the model ’ s input type can impact transferability . For example , audio inputs are time-varying and one-dimensional . They do not contain topographical structures . In contrast , images are two-dimensional , topographical inputs with neighborhoods of pixels representing a single object ( e.g. , pixels representing a dog ’ s snout ) . As a result , the input type will determine the model ’ s decision boundaries , which can impact transferability . 2.3.2 MFCC : . We hypothesize that the MFCC algorithm could be a factor limiting transferability . This algorithm uses hand-crafted filters to regularize the feature vector and consequently , remove the high spectral noise ( Mannell , 1994 ) . Since regularization of the decision boundary or gradients has been shown to impact transferability ( Demontis et al. , 2019 ; Zhou et al. , 2018 ; Wu et al. , 2018 ; Wu & Zhu , 2020 ) , we hypothesize that regularization of the feature vector using the MFCC plays a role as well . 2.3.3 RNN : . The network architecture can impact transferability as well . For example , the dropout layer , which is often included in model architectures for performance purposes , also inadvertently improves the transferability of adversarial samples ( Demontis et al. , 2019 ) . We hypothesize that the RNNis a factor limiting transferability in ASRs . This is because there are several RNN-based applications where authors have failed to demonstrate targeted transferability ( Abdullah et al. , 2021b ) .While these papers show transferability is hard in their respective domains , they do not contain definitive experiments studying why this is the case . 2.3.4 Output Type : . Since we know that output type can impact the transferability rate ( Wei et al. , 2018 ) , we hypothesize that a sequential output type ( instead of a single output label ) can influence it as well . This type of output is commonly found in ASRs . For an adversarial sample to transfer , the victim ASR needs to assign the attacker chosen character to each frame of the adversarial audio . This might increase the chances of a mistake , which for even a few frames , could lead to a semantically wrong transcript ( e.g. , “ cccaaatttt ” vs “ hhhaaatttt ” ) . 2.3.5 Vocabulary and Sequence Sizes : . The last factor we consider is the output complexity or the total number of output labels . Transferability becomes harder as the number of output labels increases ( Demontis et al. , 2019 ; Wu et al. , 2018 ; Wu & Zhu , 2020 ; Kurakin et al. , 2016b ) . However , the output labels are not fixed in ASRs , as they are in the case of images ( e.g. , 10 output labels for the MNIST dataset ) . This is because ASRs are designed to output variable size sequences to account for the variation in the input audio length . As a result , instead of considering total output labels , we use two metrics to capture output complexity : vocabulary and sequence sizes ( Figure 1 ( e ) and ( f ) ) . The vocabulary size is the number of unique words in the training data that the ASR has learned to recognize . Similarly , the sequence size is the average number of words in each audio sample in the dataset . Both of these variables work in tandem to account for the ASR ’ s output complexity . 3 Study Design . Having outlined the six potential factors , we can now design an ablation study to measure their impact on transferability . Designing this study is non-trivial since we need to account for the 11 known factors from existing literature ( Section 2.1 ) that limit transferability . If ignored , these factors alone can eliminate the transferability rate , hiding the effects of the aforementioned potential factors . We control for each of these 11 in our design . Initially , we ran our ablation study on DeepSpeech , a real-world ASR which is commonly used in the adversarial research community . However , our experiments consistently yielded 0 % transferability no matter what we did : removing or changing any of the ASR components did not change the transferability rate ( Appendix A.2. ) . Upon further investigation , we realized this was primarily due to the large complexity of the model ( a few million weights ) and training data , which forced the transferability rate to remain unchanged 0 % . Overcoming this complexity was one of the major challenges of our work . We had to carefully design our experiments to remove the impact of complexity that comes with realworld ASRs , while simultaneously exposing the hidden factors limiting transferability . As a consequence , we had to run our study on simple yet realistic ASR designs to uncover the factors impacting transferability . Dataset : Since transferability becomes harder with the total number of output labels ( Demontis et al. , 2019 ; Wu et al. , 2018 ; Wu & Zhu , 2020 ; Kurakin et al. , 2016b ) , we use the small Google Speech Commands dataset ( Warden , 2018 ) . This consists of clean , short audio files , each containing one of 30 unique labels . Each audio file is one second long ( or a vector of size ∼16,000 ) , either containing a number or a word . For the control experiment , we only choose a subset of the labels , specifically the numbers one to nine ( a total of nine labels ) . Model Architecture : With this data , we can now train a simple number recognition ASR . We use the same architecture ( Figure 1 ) typically found in existing adversarial research papers ( Abdullah et al. , 2021b ) The model outputs one of nine labels for a single audio . Additionally , we account for the known factors that hinder transferability . To do so , we train a small model of approximately 250,000 trainable parameters and do not include regularization ( e.g. , dropout ) to limit the effects of complexity and regularization . To reduce model complexity further , instead of using complex RNN cell types ( e.g. , LSTMs ( Gers et al. , 1999 ) and GRUs ( Chung et al. , 2014 ) ) , we use the vanilla RNN cell ( sim , 2021 ) ( referred to as RNN in the remainder of the paper ) . We train five instances of the ASR on the exact same setup and hyper-parameters ( architecture , random seed , epochs , batch size , training data slice , etc ) . Each of these is trained to the standard real-world ASR accuracy . Attack Formulation : For our experiment , we can choose from several existing audio domain optimization attacks . Since some of these attacks are architecture specific ( Abdullah et al. , 2020 ) , they have the potential to bias our results . As a consequence , we formulate a generic optimization attack that captures the intuition of existing works . This helps extend our results to all existing audio domain optimization attacks . To do so , we first outline the steps used by optimization attacks , which generally follow the same approach . First , a perturbation δ is produced by minimizing : ||δ| |22 + l ( x+ δ , t ) ( 1 ) where l is the loss function , x is the original audio sample , and t is the target label . Next , δ is clipped by magnitude factor α ∈ R to control the quality of the adversarial audio : δ = clip ( δ , α , −α ) ( 2 ) Lastly , the δ is added to the original audio x and clipped to create a valid sample xadv : xadv = clip ( x + δ , 1 , −1 ) ( 3 ) The clip step in Equation 2 is used to control the quality of the adversarial audio and is often specific to the target model architecture . For example , attacks that clip spectral gradients ( Qin et al. , 2019 ) can not work against end-to-end ASRs that do not have a spectrum generation step ( Abdullah et al. , 2020 ) . Therefore , in our generic attack , we remove Equation 2 making it model agnostic . Adversarial Audio Generation : We use this generic optimization attack to create adversarial samples for each of the five ASRs . Following prior work ( Abdullah et al. , 2021b ; Liu et al. , 2016 ) , we only perturb audio samples that all of the ASRs transcribed correctly . We attack every label in the dataset since some sample labels could be easier to perturb and some are easier to transfer ( Carlini et al. , 2019 ) . For example , we perturb an audio sample containing the label one to produce each of the remaining labels two to nine . Running the experiments in this exhaustive manner allows us to generalize our findings . We run the attack for 500 iterations . We save the adversarial sample every 50 iterations since the number of attack iterations can impact transferability ( Dong et al. , 2018 ) . We also ensure that every saved adversarial sample has confidence greater than 0.99 because lower confidence adversarial samples are less likely to transfer ( Abdullah et al. , 2021b ) . In total , we create 4050 adversarial audio samples . Transferability : Having created the adversarial samples for each of the five ASRs , we transfer them to the remaining four models . Targeted transferability is successful if both the surrogate and the target ASRs output the same target label . However , adversarial samples exhibit asymmetry ( Wu et al. , 2018 ; Wu & Zhu , 2020 ) , which is when an adversarial sample generated for model A transfers to model B , but not the other way around . To account for this , we average the number of successfully transferred samples between all five models . This final average is the transferability rate .

Summary: This paper presents an evaluation and explanation of the limited adversarial transferability in the ASR system. It first lists 11 known factors, e.g., smoothness of gradients, and then proposes four potential factors that limit the transferability. Then it examines the transferability in the DeepSpeech system and found its model is too complex for adversarial transferability. Hence, the author implements 5 ASR systems based on a simple model trained from the Google Speech Commands dataset. It generates 500 adversarial audio samples for each of the five ASRs and tries to transfer samples to others. The evaluation examines the target transferability rate of optimization attacks by one factor at a time. Then it gives some insights into the impact of each of the potential factors.