Targeted Adversarial Examples for Black Box Audio Systems

The application of deep recurrent networks to audio transcription has led to impressive gains in automatic speech recognition ( ASR ) systems . Many have demonstrated that small adversarial perturbations can fool deep neural networks into incorrectly predicting a specified target with high confidence . Current work on fooling ASR systems have focused on white-box attacks , in which the model architecture and parameters are known . In this paper , we adopt a black-box approach to adversarial generation , combining the approaches of both genetic algorithms and gradient estimation to solve the task . We achieve a 89.25 % targeted attack similarity after 3000 generations while maintaining 94.6 % audio file similarity . 1 INTRODUCTION . Although neural networks have incredible expressive capacity , which allow them to be well suited for a variety of machine learning tasks , they have been shown to be vulnerable to adversarial attacks over multiple network architectures and datasets Goodfellow et al . ( 2014 ) . These attacks can be done by adding small perturbations to the original input so that the network misclassifies the input but a human does not notice the difference . So far , there has been much more work done in generating adversarial examples for image inputs than for other domains , such as audio . Voice control systems are widely used in many products from personal assistants , like Amazon Alexa and Apple Siri , to voice command technologies in cars . One main challenge for such systems is determining exactly what the user is saying and correctly interpreting the statement . As deep learning helps these systems better understand the user , one potential issue is targeted adversarial attacks on the system , which perturb the waveform of what the user says to the system to cause the system to behave in a predetermined inappropriate way . For example , a seemingly benign TV advertisement could be adversely perturbed to cause Alexa to interpret the audio as “ Alexa , buy 100 headphones. ” If the original user went back to listen to the audio clip that prompted the order , the noise would be almost undetectable to the human ear . There are multiple different methods of performing adversarial attacks depending on what information the attacker has about the network . If given access to the parameters of a network , white box attacks are most successful , such as the Fast Gradient Sign Method Goodfellow et al . ( 2014 ) or DeepFool Moosavi-Dezfooli et al . ( 2015 ) . However , assuming an attacker has access to all the parameters of a network is unrealistic in practice . In a black box setting , when an attacker only has access to the logits or outputs of a network , it is much harder to consistently create successful adversarial attacks . In certain special black box settings , white box attack methods can be reused if an attacker creates a model that approximates the original targeted model . However , even though attacks can transfer across networks for some domains , this requires more knowledge of how to solve the task that the original model is solving than an attacker may have Liu et al . ( 2016 ) ; Papernot et al . ( 2016 ) . Instead , we propose a novel combination of genetic algorithms and gradient estimation to solve this task . The first phase of the attack is carried out by genetic algorithms , which are a gradient-free method of optimization that iterate over populations of candidates until a suitable sample is produced . In order to limit excess mutations and thus excess noise , we improve the standard genetic algorithm with a new momentum mutation update . The second phase of the attack utilizes gradient estimation , where the gradients of individual audio points are estimated , thus allowing for more careful noise placement when the adversarial example is nearing its target . The combination of these two approaches provides a 89.25 % average targeted attack similarity with a 94.6 % audio file similarity after 3000 generations . 1.1 PROBLEM STATEMENT . Adversarial attacks can be created given a variety of information about the neural network , such as the loss function or the output probabilities . However in a natural setting , usually the neural network behind such a voice control system will not be publicly released so an adversary will only have access to an API which provides the text the system interprets given a continuous waveform . Given this constraint , we use the open sourced Mozilla DeepSpeech implementation as a black box system , without using any information on how the transcription is done . We perform our black box targeted attack on a model M given a benign input x and a target t by perturbing x to form the adversarial input x′ = x+ δ , such that M ( x′ ) = t. To minimize the audible noise added to the input , so a human can not notice the target , we maximize the cross correlation between x and x′ . A sufficient value of δ is determined using our novel black box approach , so we do not need access to the gradients of M to perform the attack . 1.2 PRIOR WORK . Compared to images , audio presents a much more significant challenge for models to deal with . While convolutional networks can operate directly on the pixel values of images , ASR systems typically require heavy pre-processing of the input audio . Most commonly , the Mel-Frequency Cepstrum ( MFC ) transform , essentially a fourier transform of the sampled audio file , is used to convert the input audio into a spectogram which shows frequencies over time . Models such as DeepSpeech ( Fig . 2 ) use this spectogram as the initial input . In a foundational study for adversarial attacks , Cisse et al . ( 2017 ) developed a general attack framework to work across a wide variety of models including images and audio . When applying their method to audio samples , they ran into the roadblock of backpropagating through the MFC conversion layer . Thus , they were able to produce adversarial spectograms but not adversarial .wav files . Carlini & Wagner ( 2018 ) overcame this challenge by developing a method of passing gradients through the MFC layer , a task which was previously proved to be difficult Cisse et al . ( 2017 ) . They applied their method to the Mozilla DeepSpeech model , which is a complex , recurrent , character-level network that can decode translations at up to 50 characters per second . With a gradient connection all the way to the raw input , they were able to achieve impressive results , including generating samples over 99.9 % similar with a targeted attack accuracy of 100 % . While the success of this attack opens new doors for white box attacks , adversaries in a real-life setting commonly do not have knowledge of model architectures or parameters . Alzantot et al . ( 2018 ) have demonstrated that black-box approaches for targeted attacks on ASR systems are possible . Using a genetic algorithm approach , they were able to iteratively apply noise to audio samples , pruning away poor performers at each generation , and ultimately end up with a perturbed version of the input that successfully fooled a classification system . This attack was conducted on the Speech Commands classification model Alzantot et al . ( 2018 ) , which is a lightweight convolutional model for classifying up to 50 different single-word phrases . Extending the research done by Alzantot et al . ( 2018 ) , we propose a genetic algorithm and gradient estimation approach to create targeted adversarial audio , but on the more complex DeepSpeech system . The difficulty of this task comes in attempting to apply black-box optimization to a deeply- layered , highly nonlinear decoder model that has the ability to decode phrases of arbitrary length . Nevertheless , the combination of two differing approaches as well as the momentum mutation update bring new success to this task . 1.3 BACKGROUND . Dataset For the attack , we follow Carlini & Wagner ( 2018 ) and take the first 100 audio samples from the CommonVoice test set . For each , we randomly generate a 2-word target phrase and apply our black-box approach to construct an adversarial example . More details on evaluation can be found in section 3 . Each sample in the dataset is a .wav file , which can easily be deserialized into a numpy array . Our algorithm operates directly on the numpy arrays , thus bypassing the difficulty of dealing with the MFC conversion . Victim model The model we attack is Baidu ’ s DeepSpeech model Hannun et al . ( 2014 ) , implemented in Tensorflow and open-sourced by Mozilla.1 Though we have access to the full model , we treat it as if in a black box setting and only access the output logits of the model . In line with other speech to text systems Chiu et al . ( 2017 ) ; Cisse et al . ( 2017 ) , DeepSpeech accepts a spectrogram of the audio file . After performing the MFC conversion , the model consists 3 layers of convolutions , followed by a bi-directional LSTM , followed by a fully connected layer . This layer is then fed into the decoder RNN , which outputs logits over the distribution of output characters , up to 50 characters per second of audio . The model is illustrated in figure 2 . Connectionist temporal classication While the DeepSpeech model is designed to allow arbitrary length translations , there is no given labeled alignment of the output and input sequences during training time . Thus the connectionist temporal classication loss ( CTC Loss ) is introduced , as it allows computing a loss even when the position of a decoded word in the original audio is unknown Carlini & Wagner ( 2018 ) . DeepSpeech outputs a probability distribution over all characters at every frame , for 50 frames per second of audio . In addition to outputting the normal alphabet a-z and space , it can output special character . Then CTC decoder C ( · ) decodes the logits as such : for every frame , take the character with the max logit . Then first , remove all adjacent duplicate characters , and then second , remove any special characters . Thus aab b will decode to abb Carlini & Wagner ( 2018 ) . As we can see , multiple outputs can decode to the same phrase . Following the notation in Carlini & Wagner ( 2018 ) , for any target phrase p , we call π an alignment of p if C ( π ) = p. Let us also call the output distribution of our model y . Now , in order to find the likelihood of alignment π under y : Pr ( p|y ) = ∑ π|C ( π ) =p Pr ( π|y ) = ∑ π|C ( π ) =p ∏ i yiπ as noted by Carlini & Wagner ( 2018 ) . This is the objective we use when scoring samples from the populations in each generation of our genetic algorithm as well as the score used in estimating gradients . 1 https : //github.com/mozilla/DeepSpeech Greedy decoding As in traditional recurrent decoder systems , DeepSpeech typically uses a beam search of beam width 500 . At each frame of decoding , 500 of the most likely π will be evaluated , each producing another 500 candidates for a total of 2500 , which are pruned back down to 500 for the next timestep . Evaluating multiple assignments this way increases the robustness of the model decoding . However , following work in Carlini & Wagner ( 2018 ) , we set the model to use greedy decoding . At each timestep only 1 π is evaluated , leading to a greedy assignment : decode ( x ) = C ( argmax π Pr ( y ( x ) |π ) ) Thus , our genetic algorithm will focus on creating perturbations to the most likely sequence ( if greedily approximated ) .

Summary: In "Targeted adversarial examples for black box audio systems" the authors look at an adversarial problem in neural nets for audio processing. There is quite a lot of recent interest in adversarial problems in machine learning. That work is mostly on the image side, and so this work is very topical. The problem is to modify an audio signal without changing how it sounds to the human ear, so that it is interpreted as the attacker wishes by the neural network. In the black box approach, the weights of the neural network are not known by the attacker. The attacker however must be able to present modified audio and learn the network's interpretation as often as the attacker wants. This work is very exciting and topical, and of interest to the ICLR community.