Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this work, the authors study the task of building neural network classifiers for audio tasks which can be certified as being resistant to an adversarial attack. One of the contributions of this work is the development of abstract transformers which can be used for the data processing frontend used in typical audio applications. The work also proposes an abstract transformers for LSTMs which is stated to be much faster to use in practice than previous work. 

Overall, this work is interesting and I think it would be a great addition to the conference. The paper is generally well written in the initial sections, and the main ideas are very clearly presented. However, there are a number of missing details, particularly in the final sections which discuss the experimental validation. In its present form, I am rating this work as “weak reject”, but I would increase my scores if the authors can improve the final sections in the revised draft.

Main Comments:
1. While I found section 3 to be useful to get an intuition of the proposed method, I still feel that it could be condensed a bit to add in additional details. For example, the authors don’t describe “back-substitution” in the work, which I believe should be described in the main text.
2. A clarification question: When computing provability, the authors state that “We randomly shuffled the test data and then, for every experiment, inferred labels one by one until the number of correctly classified samples reached 100. We report the number of provably correct samples out of these 100 as our provability.” How sensitive was the provability metric to the choice of these 100 test examples? Was the metric computed by repeatedly sampling 100 test cases, for example?
3. The section on “Provable defense for audio classifiers” was not very clear to me. The authors state that “To train, we combine standard loss with the worst case loss obtained using interval propagation.” I was not clear on what the modified loss is. Could the authors please clarify this in the text, preferably a mathematical formulation? Also, I’m curious why these experiments are only conducted on the FSDD set, but not on the GSC set. 
4. Figure 5c. Why does the interval analysis technique perform so much worse on the GSC set relative to the FSDD set?  On a related note, it would also be useful to describe some more details about the model architectures for the two tasks.
5. The section on “Experimental comparison with prior work” similarly left me with a number of questions. The authors mention that “We found that, in practice, optimization approach used by POPQORN produces approximations of slightly smaller volume than our LSTM transformer (although non-comparable).” Could these be quantified and reported in the paper. Also, why are the approximation volumes not comparable between the two systems. 

Minor comment: It is true that most works in audio classification and speech recognition use processed frontend features such as MFCCs. However, there is also a significant body of work which operates directly on the time-domain signal. Perhaps it would be better to clarify this in the text?
For example:
Pascual S, Bonafonte A, Serra J. SEGAN: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452. 2017 Mar 28.
Sainath TN, Weiss RJ, Senior A, Wilson KW, Vinyals O. Learning the speech front-end with raw waveform CLDNNs. In Sixteenth Annual Conference of the International Speech Communication Association 2015.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents an end-to-end neural network verifier that is specially designed for audio signal processing to certify the robustness of a system when facing noise perturbation.  The approach is based on abstract transformers to deal with non-linearity in the audio signal processing pipeline and LSTM acoustic model. The authors implement the approach in a so-called "deep audio certifier" system and conduct experiments on various datasets and network architectures.  The results seem to be supportive.   The idea is good and the mathematical derivation is meticulous (although appears to be a bit tedious).  This is an interesting paper globally but I have some concerns. 

1.  There should be a more thorough introduction on how to verify the robustness of a neural network classifier for noise perturbation.  There should be some background summary such as what are the existing approaches and how to measure the robustness of a neural network classifier, etc..

2. The so-called audio processing pipeline here is actually speech processing pipeline. I am not sure if "audio" is the right term in a strict sense. 

3.  This paper is closely related to the POPQORN paper. So it is good to see some in-depth discussion and comparison between the two.  One thing that is not clear to me is that the authors claim POPQORN is very time-consuming but DAC is much faster.  I wonder if the authors can elaborate a bit more on this issue.   What exactly makes POPQORN time-consuming in this case? 

P.S. rebuttal read.  I will stay with my score. 
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes an approach for the certification of speech classification neural networks against adversarial perturbations. The network is based on a simple pipeline starting from MFCC to make an utterance level classifier via a last hidden state of an LSTM acoustic model. This approach can perform analysis through this pipeline. I feel that this paper is very difficult to follow because of the lack of background technique explanations based on neural network certification, and the lack of technical surveys of speech recognition and related areas. This paper requires such major restructuring and more surveys to make it in good shape.

Comments
- The authors only list CTC related techniques as state-of-the-art ASR, but state-of-the-art ASR is still based on the HMM/DNN hybrid system or attention-based encoder-decoder/RNN transducer. They seriously lack the surveys o this area. Also several technical terminologies are not common in the speech recognition are (e.g., automated speech recognition --> automatic speech recognition)
- "Additionally, audio systems typically use recurrent architectures (Chiu et al., 2017)": There are a lot of state-of-the-art ASR systems including TDNN (Kaldi), CNN, and transformer. Again the paper does not have enough surveys.
- The font size of the characters in figure 1 is too small.
- I cannot understand why the paper uses MFCC. The community was already moved from MFCC to log mel filterbank. We don't need final DCT.
- Section 2. Threat model: what kind of noises are using?
- Page 3, power operation: Either side must be a conjugate to get the power spectrum.




review_title: Official Blind Review #1

Meta-review: The paper developed log abstract transformer, square abstract transformer and sigmoid-tanh abstract transformer to certifiy robustness of neural network models for audio. The work is interesting but the scope is limited. It presented a neural network certification methods for one particular type of audio classifiers that use MFCC as input features and LSTM as the neural network layers. This thus may have limited interest to the general readers. 

The paper targets to present an end-to-end solution to audio classifiers. Investigation on one particular type of audio classifier is far from sufficient. As the reviewers pointed out, there're large literature of work using raw waveform inputs systems. Also there're many state-of-the-art systems are HMM/DNN and attnetion based encoder-decoder models. In terms of neural network models, resent based models, transformer models etc are also important. A more thorough investigation/comparison would greatly enlarge the scope of this paper. 
Decision: Reject