Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper applies Bayesian optimisation (BO), a sample efficient global optimisation technique, to the problem of finding adversarial perturbation. First, the application is starightforward application of BO to the well-known problem of adversarial perturbation. Nothing innovative. Second, the paper addresses the high dimensional optimisation with simple upsampling technique like nearest-neighbour, without even trying their hands dirty by using one of many many high-dimensional Bayesian optimisation algorithm (a quick Google search will reveal them), The work  thus fail in thoroughness also. Third, adversarial perturbation are known to exist even around the image such that even a simple gradient descet optimisation starting from the target image would be able to provide perceptually small perturbation (it does not have to the smallest to be perceptually small). Hence, the impact is also missing. Thus accroding to me this paper is not good enough for acceptance. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposed a BO-based black-box attack generation method. In general, it is very well written and easy to follow. The main contribution is to combine BO with dimension reduction, which leads to the effectiveness in generating black-box adversarial examples in the regime of limited queries.  However, I still have some concerns about this paper. 

1) The benefits of BO? It seems that the step of dimension reduction is crucial to make BO scablable to high-dimensional problems. I wonder if the gradient estimation-based attack methods can apply the similar trick and yield the similar performance. That is, one can solve problem min_{\delta} attack_loss( x, y, g(\delta) ) by estimating gradients via finite-difference of function values, where g(\cdot) is the dimension-reduction operator, and \delta is the low-dimensional perturbation. Such a baseline is not clear in the paper, and the comparison with (Tu et al., 2019) is not provided in the paper. 

2) Moreover, in experiments, it seems that only query-efficiency was reported. What about distortion-efficiency for BO-based attack? For $\ell_\infty$-attacks, the other $\ell_p$ norms can be used as distortion metrics. I wonder what perturbation does the BO method converge to. It was shown in (https://arxiv.org/pdf/1907.11684.pdf, Table 1) that BO usually leads to larger \ell_1 and \ell_2 distortion. 

3) It might be useful to show the convergence of BO in terms of objective value versus iterations/queries. This may give a clearer picture on how BO works in the attack generation setting. 

4) Minor comment: In related work "Bayesian optimization has played a supporting role in several methods,
including Tu et al. (2019), where ...." However,  Tu et al. (2019) does not seem using BO and ADMM. 

############ Post-feedback ##########
Thanks for the clarification and the additional experiments. I am satisfied with the response, and have increased my score to 6.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper presents an idea on making adversarial attack on deep learning model. Since the space of input-output for the adversarial attach is huge, the paper proposes to use Bayesian optimization (BO) to sequentially select an attack.

Although the potential application of adversarial attack on deep learning model is interesting, the paper contribution and the novelty are limited giving the fact that there is another related paper published [1].

The authors in [1] consider using Bayesian optimization to make adversarial attack for model testing. In particular, they have considered the deep learning model. Then, they extend to multi-task settings. There is a big overlapping between the idea in [1] and the current paper.

The paper presentation and writing is high quality although the paper is a bit over-length.

[1] Gopakumar, Shivapratap, et al. "Algorithmic assurance: an active approach to algorithmic testing using Bayesian optimisation." Advances in Neural Information Processing Systems. 2018.


review_title: Official Blind Review #2

Meta-review: The paper proposes a Bayesian optimization approach to creating adversarial examples. The general idea has been in the air for some years, and over the last year especially there have been a number of approaches using BayesOpt for this purpose. Reviewers raised concerns about differences between this approach and related work, and practical challenges in general for using BayesOpt in this domain (regarding dimensionality, etc.). The authors provided thoughtful responses, although some of these concerns still remain. The authors are encouraged to address all comments carefully in future revisions, which a sufficiently substantial that the paper would benefit from additional review.



Decision: Reject