Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors identify two issues with current certified robustness training approaches: Weight initialisation leading to exploding bounds and imbalance in ReLU activation states. They propose three improvements to mitigate these issues and cut down training time: new weight initialisation (IBP initialisation), batch normalisation after every layer and custom regularisation to encouraging tight bounds and balance ReLU activations. The approach is evaluated on MNIST, CIFAR-10 and TinyImageNet.
main_review: Overall the paper is well written and easy to understand. The investigated problem is well motivated and the solutions are natural. Further, improving and speeding up certified training is an important problem. While the approach and techniques used in the paper are reasonably simple they are quite elegant and further seem to be  effective, as shown in the experimental evaluation.  

A general question i had while reading the paper, was wether the introduced methods could be applied or adapted for CROWN-IBP, since  similar gains might be achievable there. 

While reading Sec. 3, i expected even bigger improvements compared to the reported results in Sec. 4, when IBP initialisation is used compared to standard initialisation as an exponential blow-up was mitigated. Could the authors report the numbers in Table 4, where no Batch-Normalisation is used but IBP initialisation with and without the regularisers is used?

Further Comments:
- Figure 3 & 4: The font size for the labels (axes) could be improved
- Appendix D.2: Equation after line 616: the brackets do not match. 
- The authors may want to highlight the differences to other works considering problems with current certified robustness training methods [A, B]. 

[A] Lee et.al.: “Loss Landscape Matters: Training Training Certifiably Robust Models with Favorable Loss Landscape”, 2021

[B] Jovanovic et.al.: “Why Tighter Relaxations May Hurt Training”, https://arxiv.org/abs/2102.06700
limitations_and_societal_impact: Societal Impact was not discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 

In this paper, the authors propose a  weight initialization scheme and regularizers for certified robust training.  Moreover, the authors argue the benefit of Batch Normalization (BN) in certified training. 
main_review: 
Pros.
 1. The idea of keeping the difference gain close to one to achieve stable bounds is interesting.  
2. The paper focuses on certified robust training, which is an important direction in robust deep learning.


Cons.
1. Some symbols are confusing.   In Eq.(4),  |\boldsymbol{W}_i| seems to be a vector. But in Line 171 E(|\boldsymbol{W}_i|) is a scalar.  How to achieve  E(\Delta_i) = n_i E(|\boldsymbol{W}_i|) E(\delta_i) in Line 132?
2. In table 2, the error is around 80%. In Figure 5, the classification error is larger than 85%.  I think in this setting, the performance is too weak to be practically useful. It is unconvinced to test in this extreme setting.  Thus, the empirical study on the only large dataset (TinyImagenet) cannot support the conclusion. 

Minor:
Typos.  "last year" in Line 118. 


==============================================================

==================Review  Update  =============================== 

Thanks for the authors' detailed feedback. Most of my concerns have been clarified.   So I decided to raise my score to 6.  In addition,  it seems that $\mathbb{E} \Delta_i$ is determined by the difference of the worst-case samples on the boundary.  The empirical estimation $\\mathbb{E} \widehat{\Delta}_i$ using a mini-batch may have a large variance.  It is better to verify the variance of  $\\mathbb{E} \widehat{\Delta}_i$  w.r.t the batch-size. 






limitations_and_societal_impact: The limitations are stated.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims at improving upon the training time of existing certified robust training methods. Existing methods suffer from long warm-up time needed to make the training dynamics of robust training methods stable. Specifically, the authors identify two drawbacks in existing methods, viz, poor parameter initialization and an imbalance in hidden layers' ReLU activations, both of which are shown to make the training dynamics of certified robust methods unstable. The authors propose several fixes to the existing Interval Bound Propagation (IBP) including a weight initialization, regularizations and recommend using batch norm in all layers to stabilize IBP training thereby making the warmup phase of training shorter. They show that their method achieves competitive standard/verified accuracy while being able to train significantly faster.




main_review: 
strengths:
- paper is clearly written and the ideas were easy to follow
- the proposals made by the authors were explained in detail and immediately made sense
- experiments show substantial improvement in training speed using the proposed technique
- the ablation analysis clearly shows the importance of each of the proposed components that are added to vanilla IBP

weakness:
- the proposed method introduces regularization terms which requires the regularization coefficient lambda in Eq 10. Since the main claim is regarding the speedup in training time, I think a discussion on how sensitive the performance is w.r.t. lambda would be informative. If the performance is too sensitive and extensive tuning is required, then it defeats the purpose of training time speedup. For instance, if we even perform a grid search over 5 values, the combined training time for the proposed method on CIFAR-10 (based on Table 3) would be roughly 47000 sec, which would not be a fair comparison with the baselines. Ideally I would like to see something like a table with a computational budget and the best performance along with training time of the proposed method.
- deriving initialization schemes for non-MLP (and CNN) type architectures such as ResNets requires additional considerations and cannot be trivially extended from MLP/CNN style weight initialization schemes (cf [1], [2]). Since the authors use ResNet architecture in this paper, I think that while the proposed initialization improves training speed, it may still not be the ideal initialization. This should at least be discussed in the paper.
- In section 3.3.2, it is mentioned that BN accelerates convergence by reducing internal covariate shift. This is an outdated view. More recent research suggest BN instead improves the smoothness of the loss landscape (cf [3]).

Overall I like the paper. If the authors address my concerns, I will keep my score.

Minor comments:
- Line 43: grammatical error. "can be exploded" should be "explode"
- Line 48: "during the training start" should be "at the start of training"

[1] Hanin, B., & Rolnick, D. (2018). How to start training: The effect of initialization and architecture. arXiv preprint arXiv:1803.01719.

[2] Arpit, D., Campos, V., & Bengio, Y. (2019). How to initialize your network? robust initialization for weightnorm & resnets. arXiv preprint arXiv:1906.02341.

[3] Santurkar, S., Tsipras, D., Ilyas, A., & Mądry, A. (2018, December). How does batch normalization help optimization?. In Proceedings of the 32nd international conference on neural information processing systems (pp. 2488-2498).
limitations_and_societal_impact: See main review for limitations. I think given the paper is about adversarial robustness, a short discussion on societal impact would be appropriate.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work analyzes the possible reasons why extremely large warmup and training epochs are necessary in former IBP training methods. Then it proposes a new training method for fast certified robust training. Comparing with its counterparts, the proposed method can obtain SOTA performance within much fewer training epochs. The experiments prove the efficiency of the proposed method.

main_review: Originality: The proposed method is kind of incremental (BN + two additional loss); the analysis on the concerned issues does provide some interesting insights, but the motivations/intuition of proposed method are not reasonable.

significance: The performance of proposed method outperforms its counterparts with a clear margin, but the improvement is kind of minor (performance shown in table 1, 2, with same short schedule).

Quality: The experiment results shown in the paper are clear and full of details, it is very convenient for me to assess the capacity of the proposed method. But the analysis part is unsatisfactory.

Detailed comments: I think the most important contribution of this work is to explain why former IBP training methods require extremely large training epochs, and design appropriate strategies to solve it. As far as I am concerned, the interpretations on issues (Sec 3.2) are reasonable. But the motivation to use BN (section 3.3.2) is not convincing, because it will destroy the foundation of interval bound propagation framework: note according to section 3.2.1, the interval bound can be estimated because the network only consists of linear transformation and monotone activation function. The derivation in section 3.2.1 cannot be held in normalized network. I think this might be the core reason why previous works do not utilize BN too much. Since BN is an important part of the proposed method, the author should take the effect of BN into account when demonstrate their intuition of the method. Besides, in section 3.3.2, the authors states BN is beneficial to IBP mostly by improving the balance of ReLU activation, they do not discuss another characteristic of BN, accelerating training of DNN [1,2]. It is very likely that the proposed method can be trained within fewer epochs due to accelerating ability of BN rather than balancing ReLU activation. The authors should carefully review the property of BN and clarify what is the intrinsic benefits brought by BN to IBP. 

Suggestions: 1) To verify the benefits of BN, extra ablation study should be enough because BN can be dissected into two parts: centralization (minus mean, balancing ReLU activation) and unitization (divided by standard deviation). Centralization contributes to balancing effect while unitization contribute to acceleration effect. 2) But it's really important to clarify how BN affects the interval bound in the propagation network in Section 3.2.1.

Clarity: the writing is good, I enjoy reading the paper. 

[1] Santurkar S, Tsipras D, Ilyas A, et al. How does batch normalization help optimization?[C]//Proceedings of the 32nd international conference on neural information processing systems. 2018: 2488-2498.

[2] Van Laarhoven T. L2 regularization versus batch and weight normalization[J]. arXiv preprint arXiv:1706.05350, 2017.


=====

Edit after seeing author's rebuttal

After seeing the response the author's rebuttal, I'm still not satisfied with the author's interpretations on how BN affects the interval bound in the propagation network, because they only add extra empirical verifications rather than provide solid theoretical demonstrations, I think the authors need more efforts to clarify this issues in their next version no matter whether this paper is accepted. I appreciate the addition ablation study on BN, so I raise my score to 6.


limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper makes useful contributions to robust training. The authors begin by identifying several issues with current certified robustness training and next propose effective approaches to mitigating these issues. The improvement is shown to be substantial in their experiments.
Decision: Accept (Poster)