Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This work proposes CROWN-IBP - novel and efficient certified defense method against adversarial attacks, by combining linear relaxation methods which tend to have tighter bounds with the more efficient interval-based methods. With an attempt to augment the IBP method with its lower computation complexity with the tight CROWN bounds, to get the best of both worlds. One of the primary contributions here is that reduction of computation complexity by an order of \Ln while maintaining similar or better bounds on error. The authors show compelling results with varied sized networks on both MNIST and CIFAR dataset, providing significant improvements over past baselines.

The paper itself is very well written, lucidly articulating the key contributions of the paper and highlighting the key results. The method and rationale behind it quite easy to follow.


Pros:
> Show significant benefits over previous baseline with 7.02% verified test error on MNIST at  \epsilon = 0.3, and 66.94% on CIFAR-10 with \epsilon = 8/255
> The proposed method is computationally viable, with up to 20X faster than linear relaxation methods with similar. better test errors and within 5-7X slower than the conventional IBP methods with worse errors

Cons:
> Extensive experiments with more advanced networks/datasets would have been more convincing, esp. given the computation efficiency that enables such experiments
> More elaborate insights into the choice of the training config/hyper-params esp. with the choice of \K_start, \K_end across the different datasets


Other comments:
> For the computational efficiency studies, it would be helpful to provide a breakdown of the costs between the different layers and operations, to better asses/confirm that benefits of CROWN-IBP method
> Impact of other complementary techniques such a lower precision/quantization? One fo the references compared against is the Gowal et al. 2018 for the as a baseline, however, it seems to be those results were obtained on a different HW platform (TPUs - motioned in Appendix-B), with potentially different computational accuracies (BFLOAT16 ?). So, this bears to question of the impact of precision on these methods and also the computation complexity.
> Since I'm not very well versed with the current baseline and state-of-art for variable robust training of DNN, it would be good to get an additional confirmation on the validity of the used baselines.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new variation on certified adversarial training method that builds on two prior works IBP and CROWN. They showed the method outperformed all previous linear relaxation and bound propagation based certified defenses. 

Pros:
1. The empirical results are strong. The method achieved SOTA.

Cons:
1. Novelty seems small. It is a straightforward combination of prior works, by adding two bounds together.
2. Adds a new hyperparameter for tuning.
3. Lack of any theoretical insights/motivation for the proposed method. Why would we want to combine the two lower bounds? The reason given in the paper is not very convincing:

"IBP has better learning power at larger epsilon and can achieve much smaller verified error.
However, it can be hard to tune due to its very imprecise bound at the beginning of training; on the
other hand, linear relaxation based methods give tighter lower bounds which stabilize training, but it
over-regularizes the network and forbids us to achieve good accuracy."

My questions with regards to this:
(i) Why does loose bound result in unstable training? Tighter bound stabilize training?
(ii) If we're concerned with using a tighter bound could result in over-regularization, then why not just combine the natural loss with the tight bound, as natural loss can be seen as the loosest bound. Is IBP crucial? and why?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new method for training certifiably robust models that achieves better results than the previous SOTA results by IBP, with a moderate increase in training time. It uses a CROWN-based bound in the warm up phase of IBP, which serves as a better initialization for the later phase of IBP and lead to improvements in both robust and standard accuracy. The CROWN-based bound uses IBP to compute bounds for intermediate pre-activations and applies CROWN only to computing the bounds of the margins, which has a complexity between IBP and CROWN. The experimental results are verify detailed to demonstrate the improvement.

The improvement is significant enough to me and I tend to accept the paper. The results on CIFAR10 with epsilon=8/255 is so far the state-of-the-art. However, it is far from being scalable enough to large networks and datasets, which has already been achieved by randomized smoothing based approaches. On CIFAR10, it takes 32 TPU cores to train a 4-conv-layer network. Still, such an approach has the advantage of making robust inferences much more efficiently than randomized smoothing, and thus still worth further explorations.
review_title: Official Blind Review #2

Meta-review: This paper presents a method that hybridizes the strategies of linear programming and interval bound propagation to improve adversarial robustness.  While some reviewers have concerns about the novelty of the underlying ideas presented, the method is an improvement to the SOTA in certifiable robustness, and has become a benchmark method within this class of defenses.
Decision: Accept (Poster)