Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to improve the training speed and decrease the computation cost of AdvProp, which is a method that leverages the adversarial example to improve the image recognition accuracy. AdvProp uses separate batchnorm for clean and adversarial examples respectively. In this work, the proposed method Fast AdvProp reduces the computation cost by reusing some gradient computation during training. In the experiment section, Fast AdvProp demonstrates better image recognition and object detection performance under the same training budget and can be combined with existing training strategies, such as mix up. Overall, this paper proposed an efficient training strategy that can be combined with various existing data augmentation for various tasks.
main_review: Strength 
1. The motivation of the paper is solid and the paper clearly quantifies and compares the training budget between the proposed method and AdvProp. 
2. Figure 1 and Algorithm is clear and demonstrates the overall idea of the paper.
3. The paper is well organized and well presented.
4. The proposed method demonstrates better performance under the same computation cost and generalizes to various tasks.
5. The ablation study is thorough and cover different aspects of the proposed method. 

Weakness
1. While reading the text, it is unclear what is the rationale of introducing g^{noise} in Algorithm 1. The optimization of g^{noise} could be similar to g^{clean}, but instead operating on X_2 without adding noise (changing the purple block to blue block in Figure 1(b)). What is the performance under such modification? The author is suggested to provide more detail about the introduction of g^{noise}.
2. Table 4 shows the performance of various p_adv value. What is the performance for p_adv < 0.2 ?
3. The robustness for advprop is suggested to add to Table 3 for comparison.

summary_of_the_review: This paper is enjoyable to read and provides good insight to the problem of large computation cost of AdvProp. The proposed solution Fast AdvProp demonstrates good performance with less computation cost and can potentially be applied to various vision tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes Fast Advprop which is an modified implementation of adversarial propagation. Adversarial propagation aims at improving the robustness and generalization abilities of deep neural network classifiers by performing adversarial training with additional batch normalization layers that are solely used for the adversarial examples which are used during training. The paper suggests to apply different existing techniques and small modifications to increase the training speed while using AdvProp.
main_review: STRENGTHS:
- The paper is very well written and the used techniques are well explained and each of the modifications to Advprop is justified by ablation experiments.
- The author(s) do(es) a good job at counting the number of forward- and backward propagations to demonstrate the increased performance of Fast Advprop compared to Advprop.
- The overall experimental results are promising. The author(s) show(s) that Fast Advprop is able to improve the ResNet performance on some ImageNet-like datasets while keeping the training budget comparable to the budget for vanilla training. 

WEAKNESSES:
- My main criticism is that the proposed Fast AdvProp is a potpourri of existing methods (adversarial propagation (Xie et al., 2020), free adversarial training (Shafahi et al., 2019) and Shuffling BN (He et al., 2020)) and simple modifications (splitting the training data into clean images and ones for which adversarial perturbations are calculated, rebalancing the losses of clean and adversarial images, synchronizing update speed) which limits the novelty of this approach. However, each of the modifications is well justified by ablation studies and the overall experimental evaluation is convincing.
- Table 1 points out the gained/lost accuracy in comparison with the vanilla training for some of the entries. However, for most of the entries this difference is not stated. Is there a reason for this? Why are some of the differences printed in green and not in red? What does the red arrow indicate? This should be explained in the caption of the table.
- Although, the methods used in this paper are well explained and justified, I would have liked some more explanations in Algorithm 1, e.g., which line corresponds to which paragraph. "Fuse the gradients" could for example be expressed as "Rebalancing gradients" to make the connection to the relevant paragraph.

MINOR REMARKS:
- missing word on page 6: "When attack using running statistics and non-targeted attack"
- missing/wrong word on page 6: "Before feed the adversarial images on each GPU into the network" 
- page 8: "shuffling BN" vs. "Shuffling BN"
- capitalization of references: "equation 4" vs. "Table 3"

summary_of_the_review: Overall, I would recommend to accept this submission into ICLR 2022. The paper is well-written and the results are good. I would be happy to see some of the points, that I stated in my review, would be implemented in a future version of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: AdvProp is not efficient. This paper tries to overcome that with several measures: (1) decoupling, where they now do not need to have one adversarial sample with each clean image, (2) one step adversarial rather than multi-steps, and (3) reuse recent work on accelerating advprop such as recycling gradient, for which they provide several insights.
main_review: Strengths of this paper: 

Highlighted several "tricks" to save on computational cost for AdvProp. I recognized that the discovery of these tricks and that they work to a certain extent is non-trivial work and appreciate those insights. Namely, they are (a) decoupling does not hurt as much, (b) 1-step adversarial does not cause as much degradation as previously thought, (c) recycling gradient works better with (i) non-targeted attack, (ii) using batch statistic for adversarial image generation, (iii) using shuffle BN to avoid label leaks, (iv) rebalancing examples and (v) synchronizing the update speed of parameters.

The paper is also written very clearly and the motivation is well communicated.

Weaknesses of this paper:

I felt there will be questions on what's the "new scientific discovery" here. This paper is mostly a report on empirical insights, that decoupling or reducing the number of adversarial samples does not really hurt performance that much. Most of the gradient recycling discussed in the paper is prior work but with engineering insights. 

On experimental results. I take some issues with this "The AdvProp also shows compelling robustness compared with the vanilla training, verifying its effectiveness. However, the comparison is unfair as AdvProp using 7× training budget. In the same budget setting, the performance of 15-epoch AdvProp (dividing the total epochs by 7) degrades significantly." The motivation of the paper is that you can retain the advantage of AdvProp without paying the computational cost, i.e., as the authors called it, a "free lunch". If you can't retain the advantage of advprop, then I question what is the purpose.
summary_of_the_review: I am ambivalent based on what I mentioned above. The paper provides empirical insights, but IMO not huge enough for ICLR, and is well written. However, I am wondering whether there is any new scientific discovery here -- it will be nice for those "tricks" mentioned in the paper, the authors can contrast clearly why they are novel as it seems like they have been previously proposed. I also question the AdvProp vs Fast AdvProp results where we have to significantly reduce AdvProp to have the Fast version beat it. I will be interested to see what the other reviewers think and I am opened to discussions and changing my feedback.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an improved version of AdvProp, to speed up the training process. Specifically, it proposes to use less adversarial examples in a mixed batch and utilizes a list of recent fast adversarial training techniques. The experiments have been conducted on ImageNet sets with different backbones, demonstrating the effectiveness of the proposed fast model. 
main_review: The paper actually has resolved one of the main issues of AdvProp, making it more efficient. The empirical study shows very promising results. However, I still have some concerns about the work:

1. The novelty and technique strength is limited. The way to combine clean and adversarial batches is empirical. The trick used for fast adversarial training has been explored by many other previous works. 
2. The empirical study only shows the effectiveness of fast AdvProp on the ResNet family, but not EfficientNets. In addition, on the detection task, the performance improvement is not that significant. 
3. Some claims and empirical supports of the paper are confusing. For example, [chen et al. 2021a] uses an adversarial feature augmentation instead of doing something like AdvProp to generate real adversarial samples. For me, AdvProp has more advantages to improve the clean accuracy. But the paper shows some experiments on the robustness side. 

A few notice about more technical details regarding AdvProp: 1. for the TensorFlow version, PGD-5 is better than PGD-1. But the Pytorch version [a] has a reversed conclusion; 2. the two BN trick was analyzed by the paper [b], I am not sure if the author has a similar observation on that. 

a.https://github.com/tingxueronghua/pytorch-classification-advprop
b.Adversarial Masking: Towards Understanding Robustness Trade-off for Generalization
summary_of_the_review: Overall, I think the paper is interesting but has some technical details to be addressed. In addition, the novelty of the paper is increamental.
review_title: null

Meta-review: This paper improves the training speed and decrease the computation cost of AdvProp, which is a method that leverages the adversarial example to improve the image recognition accuracy. The method achieves the speedup by leveraging a collection of practical heuristics, including reusing some gradient computation during training. The paper is well written, well justified with empirical supports, and can be potentially useful in many vision tasks. On the other hand, some novelty of the method is incremental, and the issues regarding empirical results and claims pointed out by the reviewers need  to be addressed in the revision.
Decision: Accept (Poster)