Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a robustness transfer method that combines mixup data augmentation, distillation from both prediction and intermediate features. 

It is motivated by the theoretical insight that robust test error can be decomposed into adversarial loss plus distillation loss, and both of them can be upper bounded by mixup loss. Motivated by some other previous work, the KL divergence with the teacher model’s prediction and L2 distance with the teacher model’s intermediate activated feature maps are also added to the final loss of the student model. So the loss is composed of three terms: mixup loss, KL divergence between predictions, and L2 distance between the intermediate activation maps. 

Experiments on popular datasets such as CIFAR-10, CIFAR-100 and ImageNet demonstrate it’s superiority compared with a variety of baselines. 

main_review: All the techniques seem well motivated, but their combination seems very heuristic. Results seem convincing. 

My main concern is: the final loss is composed of three terms, but there is not much ablation study to understand the importance of each term. There is some ablation study in terms of adding mixup or not, but we don’t know how the later two terms have helped. 

Clarity needs improvement. 
* notations are a bit confusing. For example, in Eq. (1), what does argmax of an indicator function mean? Did you perhaps miss Prob()?
* Function $\phi$ was used as softmax, then later was again used as the activated channel map function
* there are many tables, but no highlight in each table, hard to read
* line 308: “Table 3” should be something else? 

Some questions: 
* line 152: “Compress the hypothetical space $\mathcal{F}$ into a smaller space {$ g \circ h^T, g \in \mathcal{G}$} ” does it mean $\mathcal{F}=${$g \circ h^T, g \in \mathcal{G} $}? I guess not, but I think writing in this way is a bit confusing. 
* line 274: “mixup coefficient is $\lambda=1$”? doesn’t that mean there is no mixup? 
* In Table 2 caption, how is the clean accuracy “significantly higher”? I see in the left column Free-AT achieves the highest clean accuracy. 
* line286: “random augmentation” was never introduced earlier, and suddenly it’s added to the proposed method and shows better results. Is random augmentation also added to the baselines? 

limitations_and_societal_impact: lack of understanding for each of the three terms
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The submission proposes a novel robustness transfer method, named MixACM, to transfer robustness from a robust teacher to a student by matching activated channel maps generated without expensive adversarial perturbations. Extensive experiments in cifar and imagenet show the improvements.
main_review: 1. The paper is well written and easy to follow. 
2. in Line 171, theorem 1 starts with the assumption that a fully connected neural network with ReLU or max-pooling. However, in Fig 1, the proposed method has the blocks of resnet. It is not the same with the assumption in theorem. 
3. Fig 1 also shows the intermediate features. But this also didn't appear in theorem 1. The author can give more explanations on this part. 
limitations_and_societal_impact: none
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for robustness transfer. Based on the theoretical analysis, an activated
channel maps-based distillation along with mixup is proposed to replace the traditional adversarial loss for
robustness transfer. Experiments on CIFAR10, CIFAR100, and ImageNet datasets with different settings
demonstrate the proposed method achieves outstanding performance.
main_review: 1. The writing is relatively good and the paper is easy to understand.
2. The motivation is very interesting, and the theoretical analysis well explains the principle of proposed
methods.
3. Some writing mistakes should be fixed, for example, in Line 35, ‘compare’ should be ‘compared’.
4. The proposed method seems to be a simple combination of some existing methods such as
knowledge distillation, mixup and activated channel maps. Considering that the proposed method is
essentially a distillation method, compared to previous works for distillation, this paper may lack
novelty.
5. The experimental part lacks some essential ablation studies. For example, what’s the performance of
directly distillating the middle features? The performance gain gotten from the activated channels
maps should be presented if the teacher and student use the same dataset.
limitations_and_societal_impact: Please refer to the Main Review part for detail.
I encourage the authors to clarify the novelty more clearly in the rebuttal.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims to transfer the robustness from a robust teacher model to student model with the help of mixup augmentation. It proposes the MixACM method to transfer the robustness from teacher model to student model by matching their channel-wise activation maps.  From the theoretical perspective, it showed that adversarial loss can be bounded with the natural loss and distance between the student and teacher model on mixup examples. Empirically, extensive experiments are conducted including robustness transfer, distillation and transfer learning to demonstrate the effectiveness of the method. 
main_review: Strengths:

This paper is well-written and conducts solid experiments to demonstrate the effectiveness. From Table 1, we can see that the robustness of the student model outperforms the teacher models, and is also comparable to most state-of-the-art adversarial training methods. It also compares with the latest robustness distillation methods, IGAM and RKD, it seems that it sets a new benchmark in this robustness distillation. I am glad to see that the authors evaluate the robustness under AutoAttack and also give the distillation results on ImageNet. 

Suggestions:

From Table 6 and Figure 7 in supplemental materials, it seems that the performance gained in student models mainly comes from the feature distillation. I think it would be better to give the experiments of ACM on different layers. Does the low level or high level features matter?

limitations_and_societal_impact: The paper have discussed the limitation in the supplemental materials, the robustness transfer deteriorates significantly if depth or width of student model is reduced significantly.
review_title: null

Meta-review: This paper describes a method (MixACM) for transferring robustness from a teacher model to a student mode using activated channel map matching and mix-up, obviating the need for expensive adversarial training.

A key concern raised by reviewers was that, while the individual components of the method are well motivated and the overall combination of them appears to be highly effective, these individual components are already known, and their simple combination is not as novel or theoretically motivated as one might hope. Reviewers also expressed concerns about quality of writing and the overall clarity of the paper.

This is something of a borderline case, but given the strong results produced by this method, I nonetheless recommend that the work be accepted.

Decision: Accept (Poster)