Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposed a new method for generating adversarial examples for the task of automatic speech recognition. The authors suggest using their method called Iterative Proportional Clipping, which limits the amount of change we allow at every time-step. 

Overall this paper is an incremental research work. The paper is clearly written. The idea is intuitive and well presented.

I have several questions to the authors: 
1) Can the authors provide more details regarding the attack setup? Why exactly did you run the attack in two stages? What makes the difference between the train and eval modes? Only on the batch-norm layers? 

2) All experiments were conducted in a white-box settings, did the authors try to explore gray/black box settings as well?

3) It seems like there are phase mismatch issues in the generated adversarial examples. Did the authors try to generate adversarial examples using other approximations besides the MFCC to wav approximator? Maybe working at the spectrogram magnitude level? 

4) Regarding comment (3). the underlying assumption of adversarial examples is: "it will not be distinguishable from human ears." However, the changes to the signal are highly noticeable. Did the authors try to analyze when is it more or less noticeable, under which settings? Does it depend on the target text? 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper introduces an adversarial attack mechanism, Iterative Proportional Clipping (IPC), based on a differentiable MFCC feature extractor. In contrast to other audio attacks, the authors claim that the perturbations are well masked and are much less audible. 
They also claim to propose the first attack mechanism that takes temporal dependency into account, at least in some intuitive sense as described in Figure 2. Simulations show that state of the art speech recognizers can be attacked using the approach and certain type of defenses, based on temporal structure, can be circumvented. 	The method relies on the idea of using ‘the proportionally clipped gradient’ as a direction estimate for original audio modification. The basic idea is quite sensible and seems to be effective as supported by computational experiments.

The dataset is LibriSpeech and the speech recognizer is based on wav2letter+ -- here the MFCC features are replaced with a differentiable version to allow gradient based adversarial attacks. The experimental section is detailed and the results illustrate the 

The fundamental shortcoming with the paper is that while the method is described in detail, there is not much effort in explaining and formalizing what aspects of the proposed method make it perceptually more appealing -- as this seems to be the one of the main motivations of the paper. I would at least expect to see at least some effort to explain the proposed  clipping technique as maintaining the phase properties of the original signal. The authors claim about perceptual properties of the method seem to be somewhat anecdotal and, while I think the method is sensible, just showing the power spectrum is not very informative. Is the proposed method better just because it does good detection of the  silent periods?

I also find the use of technical jargon somewhat ambiguous at occasions (such as the term ‘linear proportionality’)

Minor:

audios -> waveforms  (audio does not have a plural form ) 

Fig 2 ‘frequency - signal frequency’ labels -- this is probably just an estimate of the power spectrum by DFT magnitude

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a technique to generate audio adversarial examples based on iterative proportional clipping. It is essentially an optimization approach via backprop but with a constraint on the magnitude of perturbation in the time domain.  The authors show the effectiveness of the proposed technique to successfully attack the wav2letter+ ASR model and claim to have superior temporal dependency than other existing approaches.  Overall, the work is interesting but I have the following concerns.

1. In terms of analysis, I think it would be more helpful to show results in spectrogram rather than frequency as shown in Fig.4.   

2.  The authors claim that the proposed technique is less time-consuming to generate in the abstract. However, there is no discussion and comparison in the experiments. 

3.  Since the proposed technique is claimed to be iterative, it would be helpful to demonstrate the "iterative" performance improvement.  How much does the iteration help?  How many iterations does it need to get a good adversarial audio?  How long does it take to accomplish that?  I couldn't find such information in the paper. 

4. By listening to the provided audio samples, I would say the quality of the audios are good in general. But in w4, one can actually perceive the distortion in the audio. But overall it is good. Thanks for the samples.  

5  The wording and formulation need to get improved.  
     --  I found the word "bandwidth" very confusing especially when it comes to the discussion about the time and frequency domain signal processing.  The word "bandwidth" has a well-established meaning in signal processing already.  
     --  There is no definition of "TD" throughout the paper -> temporal dependency? 
     --  The formulation in Eq.2 is incorrect. $\delta$ in the second term seems to be independent of the first term. shouldn't it be $l(x+\delta, y')$ in the first term?  

P.S.  rebuttal read.  I will stay with my score. 
review_title: Official Blind Review #3

Meta-review: This paper proposes a method called iterative proportional clipping (IPC) for generating adversarial audio examples that are imperceptible to humans. The efficiency of the method is demonstrated by generating adversarial examples to attack the Wav2letter+ model. Overall, the reviewers found the work interesting, but somewhat incremental and analysis of the method and generated samples incomplete, and I’m thus recommending rejection.
Decision: Reject