Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a speech synthesising-based attack (SSA) framework that generates audio examples entirely from scratch to attack automatic speech recognition (ASR) systems. A conditional variational auto-encoder (CVAE) based speech synthesiser is trained based on a combined connectionist temporal classification (CTC) and regularization loss. As compared to existing methods, the proposed approach serves as the first work that generates adversarial samples without audio inputs. The results confirm that the proposed SSA framework can attack the ASR system successfully. A comprehensive analysis of different hyper-parameters to the achievable performance is provided. Moreover, demo audio samples are provided by a website link. 

main_review: (1) The main contribution of this work is to extend the previous work by replacing audio inputs with the style vector z in the CVAE speech synthesizer. The framework is new, and the results are promising. However, the scientific depth seems a bit shallow. Since ICLR is a top machine learning conference, I would suggest the authors further highlight the major contributions, especially the scientific novelty, of this work and potential impacts to related research fields.  
(2) Additional experiments should be conducted to confirm the effectiveness of the proposed approach. In previous works, such as (Carlini & Wagner, 2018), an audio input is given, and the goal is to modify the input so that human listeners cannot perceive the modifications while ASR performance is considerably affected. On the other hand, the SSA framework directly generates audio samples from the text. Thus, an additional experiment is required: using another ASR system, trained from other datasets or constructed by another model architectures (such as LAS or hybrid CTC/Attention), to recognize the adversarial audio samples synthesized by SSA. We expected to see that the adversarial audio samples can only attack the target ASR system while not affecting the other ASR systems.
(3) The authors state that “Note that the audio x generated by CVAE model G(.) has been tested to be natural sounding using the mean opinion score obtained from Amazon Mechanical Turk” Please show the results of listening tests in the paper.
(4) Since the goal of this work is to attack ASR systems, it is meaningful to show that human recognition performance is not affected. Please conduct additional experiments and include the results in the paper.
(5) Figure 7 compares different adversarial samples. Since the samples generated by (Carlini & Wagner, 2018) and SSA are from different sources, it is reasonable that the samples show very different patterns. To us, the comparison does not provide useful information. It would be more meaningful to show and compare the vector z trained from the loss of normal speech synthesis and the SSA loss.
(6) It is very nice that the authors provide demo samples on a website. By listening to the samples provided on the website, however, we note that the samples generated by SSA present perceivable distortions. The quality is worse than the ones of (Carlini & Wagner, 2018). I am not sure whether the distortions come from the speech synthesizer or SSA loss. Please comment on this or provide additional experimental results using another speech synthesizer.
summary_of_the_review: The promising results confirm the effectiveness of the proposed SSA to attack the ASR system. The authors also provide a comprehensive analysis of hyper-parameters setups to the achievable performance. Moreover, demo audio samples are provided by a website link. The paper is well-written, and the theoretical part should be correct. However, we think additional experiments should be conducted before the paper can be accepted for publication. Moreover, the authors are suggested to summarize the major theoretical contributions on this study and discuss potential impacts to related research fields
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes a white-box attack on ASR systems using TTS.  The main technique is (somewhat) standard adversarial training with the constraint that the CVAE component of a TTS model is the only component that is updated.  Since this component is responsible for prosodic and other non-segmental qualities of speech synthesis, the expectation is that the naturalness of the synthesis will not be impacted, but the ASR recognition will be modified.
main_review: The most interesting component of this paper is the fact that adversarial attacks can be constructed by manipulation of the prosodic characteristics of synthetic speech.

However, there are a number of unanswered questions about this adversarial attack.
First and foremost, it is not at all clear that the use of synthesis and prosodic manipulation is a more compelling attack than adding adversarial noise to carrier speech.  The context where there is no available carrier speech is unconvincing -- an attacker can record or find some short speech transcript as easily (if not much easier) than they can train a TTS model that can be updated by CVAE.

The white-box assumption is reasonable for this kind of proof of concept, but it's less convincing than a black-box attack.

There is no assessment of TTS quality either of the un-modified initial utterance (that should be recognized correctly by the target ASR model) or the adversarial utterance.  The adversarial examples are fairly intelligible as the original phrase, but the naturalness is substantially impaired (i.e. to my ear many are obviously synthetic speech).  This is a limitation especially in comparison to audio dependent attacks (ADA) which sound like natural speech with a small amount of background noise.  For this attack to be completely successful the synthetic audio should be indistinguishable from human speech.  This isn't assessed.

Smaller questions:
* Section 1: Please elaborate on the use of Google Assistant and Apple Siri that are 'safety-critical'
* Section 4.1: In what case is 'benign audio unavailable'
* Section 4.1: the claim that this objective function "guarantees [that] the audio x [is] natural sounding" is over stated.  TTS aims for natural sounding speech, but this is far from guaranteed.
* Section 4.2: why is dependency on some ground truth text preferable to dependency on some ground truth audio?
* Section 4.2: what is the rationale for including levenstein distance as distinct from WER?
* Section 5.1: there is no evaluation of Synthesized Audio Quality


summary_of_the_review: Technically this paper is sound. However, the contribution here is rather narrow.  It is interesting, but not especially consequential, that CVAE manipulation can be used to localize the contribution of an adversarial attack.  It is unclear that the presence or absence of a carrier speech signal in favor of a carrier tts utterance is a consequential modification of this attack.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new adversarial attack mode to ASR systems based on speech synthesis. The main idea is to synthesize speech utterances that naturally sound like ground-truth transcription y_o, but can fool an ASR system. It is a targeted attack model, where each attack signal has its own target label y_t. The paper also presents a CVAE-based speech synthesis system and a learning rate decay method for training. 
main_review: Strength:

- The paper does provide a new adversarial attack mode that is different from the audio "dependent" attack (ADA) methods. While the ADA methods are usually performed by perturbing existing audio signals to the degree that human perception doesn't notice the difference, the proposed model can synthesize new speech signals that sound totally different from what they are intended to fool the ASR models with. 

- After checking out some of the demo signals posted on the anonymized website, the reviewer found that the synthesized speech signals are indeed in good quality, i.e., they do sound natural and convey the original meaning, y_o.

- The targeted attack results are substantially better than the other models. 

Weakness:

- It is not too clear to me what is the real-world attack scenario based on the proposed method. For example, I get the idea of ADA where an attacker secretly adds inaudible perturbation noise to the user's speech, so that the ASR system fails to recognize the voice command, e.g., for voice assistant services. However, what is the point of synthesizing new utterances from the attacker's point of view? The paper does not discuss this point of view, although it must nicely justify why the synthesis-based attacks are useful. 

- There have been other adversarial models that are based on synthesized audio. For example, "hidden voice commands" [a], "inaudible voice commands" [b], and "DolphinAttack" [c] are the systems that are based on synthesized attack signals. While the latter two are to attack the system via inaudible sound (i.e., in the ultrasound frequency) and the hidden voice commands are basically noise signal which is hard to tell. I understand the difference between the proposed model and these existing models. But, the proposed model is lacking a discussion as to how the attacker can successfully use the synthesized voice, while the other afore-mentioned models are with convincing and clear attack scenarios. The paper might benefit from additional justification. 
[a] https://people.eecs.berkeley.edu/~daw/papers/audio-dls18.pdf
[b] https://www.usenix.org/conference/nsdi18/presentation/roy
[c] https://dl.acm.org/doi/10.1145/3133956.3134052

- The authors claim that the generated attack signal sounds natural and can be perceived as the ground-truth label y_o. While the reviewer also observed that the samples uploaded on the website are convincing, I believe that a large-scale evaluation on (a) sound quality of the synthesized speech (b) human annotator's transcription is necessary. The authors indeed pointed out in the conclusion section that there are unnatural examples they observed. 

- Related to the previous point, I believe that the loss function eq (3) isn't well justified. The first term is only to make sure that the produced speech signal is recognized as y_t, which is for the targeted attack part. However, the regularization part is just to make sure that the latent variables are following a standard normal distribution. Although I can see that the model is based on a CVAE voice synthesizer, and such regularization can indirectly enforce the model to still behave like a speech synthesizer as pre-trained. However, it is still not too clear if it is the best way. 

- I am curious to know how the proposed model performs in the noisy/reverberant environment. Can it still fool the ASR system if the synthesized speech itself is contaminated by real-world acoustic perturbations?

- The proposed method is based on the white-box attack. Additional discussion on the black-box attack cases my strengthen the paper. 

Other minor comments

- Definition 1 needs more elaboration. Hard to follow what the set is defined by due to the many intersection operations. 

summary_of_the_review: I found this paper novel (to my best understanding) and interesting in that it provides a new adversarial attack mode. I wish that the paper is better justified as to when this kind of attack can be a real threat, instead of relying on the reader's own conjecture. The paper is missing discussions on the comparison to the other AIA methods. The demo signals sound natural, but it might be better to include some subjective test results as the optimization on the "naturality" part isn't too clear.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work the authors explore a new vein of research in the area of audio
adversarial attacks wherein instead of focusing on perturbing existing audio,
they synthesize audio. To their best knowledge, and this reviewer, this is 
the first such attempt. To this end, the authors use a conditional
variational auto-encoder as a speech synthesis (TTS) model and develop 
a adaptive sign gradient decent algorithm in order to solve their SSA
optimization problem. Experiments are provided showing the effectiveness of 
their approach, along with a detailed analysis of results.

main_review: The paper is well written, clear, the problem stated concisely, ample references
are provided, and the experiments are convincing. This novel type of attack -
synthesizing audio instead of perturbing - is interesting as one may not
have access to ground truth audio to perturb in an attack. Additionally, while
the overall approach is in theory complicated as it involves a STT and TTS system
and optimized for a specific loss, the authors present their method clearly.

The quantitative presentation is good and the inclusion of their adaptive sign
gradient decent algorithm may be useful for further researchers.

The authors show the validity of their SSA approach by demonstrating its superior
ability to attack an ASR system. Although, as the authors state, this is maybe
not so surprising given the fast that their SSA approach is able to modify
the input in many ways, whereas the perturbation approaches are limited in
how they can modify the input.

The only minor point is that there was no MOS done on samples the authors 
generated. And, as the authors point out, and as this reviewer was able to listen,
some of the audio samples do not sound well. But, given the large scope of
the work, this can maybe be done in the future.
summary_of_the_review: Well written, novel contribution, good references, and compelling and complete experiments.
review_title: null

Meta-review: This paper introduces a way of generating adversarial speech samples to attack an ASR system based on speech synthesis.  The proposed approach, by the name of Speech Synthesizing based Attack (SSA) , does not rely on real speech to create adversasial samples but rather create them purely from text using  a conditional variational auto-encoder.  Experiments are conducted on three datasets and the results appear to support the effectiveness of the proposed approach. 

All reviewers consider the paper technically sound and well written. Overall, the work is interesting. It pursues another possibility for a threat model on ASR and may inspire related work in the community. Reviewers also raised a number of concerns most of which have been cleared by the authors' rebuttal.  However, there are two major ones standing. One is the perceptual quality of the synthesized speech and the other is the justification of the application scenarios in the real world.  In the follow-up MOS experiments, there seems to be a noticeable difference (4.09 vs 3.39) which means the synthesized speech does sacrifice quality.  From the real-world application perspective, the scenarios proposed by the authors seem to be a bit contrived.  These two drawbacks are considered significant and need further investigation and justification.
Decision: Reject