Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the author study the transfer learning setting for training a robust classifier using randomized smoothing. It relies on a pretrained smooth model, such as MACER and SmoothAdv, as the teacher model, and proposes a simple loss function to make the student model also learn a similar smoothness. With a good teacher model, the student model can be trained very quickly, much faster than training from scratch. Adequate experiments on CIFAR-10 and ImageNet are conducted with ablation studies to show the effectiveness of this approach, where the improvement is mostly on training time. Even in the setting where a teacher model is not available, the authors show that the training a smaller teacher model first can still be beneficial.
strengths_and_weaknesses: Strength:

1. Thorough empirical results with good speed up on training time, even on large datasets (ImageNet) and large models (ViT-B).
2. Method is simple, effective, and sound.
3. The paper is well presented and easy to understand.

Weakness: 

The novelty of this work is limited, because it simply combines transfer learning with robust training using a simple loss function, without further theoretical justifications. However, I feel it is solving a real problem that training a certifiably robust model can be slow, and using a teacher model can indeed speedup the training of large robust model by a large factor without sacrificing too much performance.
questions: Although this paper proposes a quite simple algorithm but I feel it is solving a real problem and the proposed approach is effective. Thus, I tend to accept this paper. I have no major questions regarding this paper.

limitations: Limitations are well discussed and I like the discussions on page 9.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method for more efficient re-training of certifiably robust models by means of knowledge distillation. Specifically, the proposed method minimizes the l2 distance between the student and teacher softmax predictions on Gaussian-noise perturbed inputs, where the teacher is a certifiably robust model trained. 
strengths_and_weaknesses: Strengths
- The paper empirically shows that the proposed method significantly speeds up training a certifiably robust model on CIFAR-10 (with the constraint of requiring access to a certifiably robust teacher model), and can scale to ImageNet. 
- The contributions and motivation of this paper are made clear.
- The writing is straightforward. 
- The distilled models have better certified robustness than those trained from scratch given the same network architecture, even when the model is a “second-generation” distillation. 
- The transfer of certified robustness using distillation is a novel application of distillation. 

Weaknesses
- The related work is lacking in terms of prior works on knowledge distillation - it would be useful for the reader to include a section on knowledge distillation and relate the technique used in this paper to existing works. 
- The experimental evaluation could be somewhat more rigorous, and the proposed method explicitly compared to the consistency regularization method referenced in section 5.4. 
questions: - How is clean accuracy affected? 

Suggestions:
- The motivation of the paper could be discussed and evaluated in more detail. For example, the scenario considered in this paper is the need to re-deploy a different architecture using the same dataset, but it would also be interesting to explore fine-tuning the student network on new data. 
- It would still be helpful to include in Tables 6 and 7 the certified robustness of the student architecture when trained from scratch in order to better calibrate the results. 
- The consistency regularization method mentioned in section 5.4 should also be referenced in the background section at the beginning of the paper. 

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: I think the paper is well-motivated and the proposed method is straightforward. This topic is interesting. However, the empirical contribution is over-claimed to me by a wrong measurement and in fact they might be not significant and lack enough experiments. I therefore vote for rejection for now but am open to changing depending on the authors’ response. 
strengths_and_weaknesses: ### Strength
The writing of this paper is pretty clean and easy to follow. The motivation is very clear and the presentation of the proposed method carries that clearness. 

### Weakness
My major concern for this paper is that the empirical contribution is over-claimed. However, Section 5.1 is the place I think the authors measure their work in a correct way but the corresponding results are neither significantly better nor comprehensive enough to support their claimed contribution. I will elaborate. 

**Training time comparisons are unfair.**  The cost of the CRT pipeline in terms of the per-epoch time is measured in the wrong way. The current way, excluding Section 5.1, to measure the training cost for CRT is total cost = student cost. If this paper is about to compare the training cost between baselines that train student networks, this is correct. However, all the baselines are methods that actually train the teacher network. Therefore, the correct cost of CRT should be *total cost = teacher cost + student cost*. It is okay to assume a robust teacher network exists but not to assume that the cost of having a teacher network is zero. The authors seem to have noticed this problem. I found the correct measurements in Section 5.1. In fact, if this paper follows the way Section 5.1 is designed, the results will be very impressive: only do robust training on a tiny/small network and use them as teachers for monster networks, which can save a significant amount of time. To do this, the current experiments in Section 5.1 is far from enough. As one may notice that the training cost of the teacher dominates the training budget of CRT because training students do not need robustness regularizations. If one ignores the training cost of teachers, this paper is just to compare the training cost between robustness regularization and standard training, which is not interesting. 

**The significance of the work needs more justifications.** All the results show that the student networks are just marginally better or worse than the teachers. This raises a question: why in practice does one want to do this robustness transfer? Why do we not directly use the teacher network? The transfer is only between architectures and not between datasets. Unless network architectures are limited in some sense or I missed something, I don’t see why we need CRT to produce a similar network. 

**The discussion on the scalability of CRT is confusing.** Section 5.3 aims to show the scalability of CRT, which is pretty confusing to me. My superficial understanding of the scalability of CRT is determined by the scalability of training teachers and the scalability of training students. The scalability of teachers is determined by the robustness training method proposed by other work. The training of student models is just standard training, which always scales up. However, if the authors take my advice to shift the focus of this paper to train large robust student networks from small teachers, then it is fair to claim that CRT is the way to scale robustness training for large networks. 

**Some figures and tables are not necessary.** I find Figure 1 and Table 1 are not necessary at all because 1) I don’t see any information related to the topic this paper tries to discuss in Figure 1. This is just a very general plot for deep learning, and 2) making the factors in a table does not help convey more messages than pure text. There is no more information at all. 


questions: 
My questions are embedded in my comments about the weakness. I can summarize them again in this part but please find them in the context above. 

1) Why is this work significant as students are no significantly better than teachers and the robustness transfer is only between architectures?

2) Why is the training cost of CRT only the cost of training students, but is used to compare with baselines that actually train teachers ? 

limitations: The paper has a separate paragraph to discuss its limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes certified Robustness Transfer (CRT) for reducing the computational overhead of any certifiably robust training method through knowledge transfer. The paper targets a scenario where different neural network architectures evolve, and one needs to retrain different architectures with time-consuming, certifiably robust training methods.
strengths_and_weaknesses: Strength:
+ The paper conducts solid experiments to demonstrate the effectiveness and efficiency of the CRT framework.

Weakness:
- The paper lacks novelty. The approach of using transfer learning to improve (certified) accuracy and training efficiency is not new. The fact that transfer learning can transfer the robustness is already explored by existing works, as mentioned in section 3.3. The paper provides a theatrical analysis of the transferability, but the analysis is trivial. 
- Another concern is that it is unclear whether the robust knowledge has been transferred to the student model apart from metrics like certified accuracy and ACR. In other words, the paper would be improved by theoretically analyzing the decay of the robustness transfer, e.g., after $r$ recursions, specific robust metrics would decay at rate $h(r)$.


Detailed comments:
1. For all tables presenting certified metrics and speeds, please show the baseline training method with data augmentation by Gaussian noise. Comparing this baseline method can help illustrate the necessity of fancy training methods and CRT.
2. The paper does not explore how far transfer learning can go without considering the cost of training, e.g., is it possible to improve algorithm 1 such that it incorporates the existing loss (capturing the knowledge of the teacher) and another loss that uses fancy certifiably robust training methods or the baseline Gaussian noise data augmentation. 


===============================

Post one round of rebuttal:

Changed Contribution from 1 to 2

Changed Rating from 3 to 4

===============================

After reading other reviewer and authors discussion about experiments 

Changed Rating from 4 to 5
questions: 1. In the recursive evaluation of CRT in table 3, the result of chain length 3 became worse than shorter chains. Does it imply that the effectiveness of CRT will decay with respect to the number of recursions? If it does decay, then it will harm the speed up brought by CRT because one needs to retrain after applying CRT two or three times.
2. Why not directly minimizing $z^{y_i}_\phi(x_i + \eta) - z^{y_i}_\theta(x_i + \eta)$ but minimizing the distance between them?
3. Algorithm 1 only minimizes $\|z^{y_i}_\phi(x_i + \eta) - z^{y_i}_\theta(x_i + \eta)\|_2$. How about incorporating other losses? 
limitations: Yes, the authors adequately addressed the limitations in section 5.5.
review_title: null

Meta-review: metareview: This work considers how to transfer a well-trained, certified robust model (i.e., randomized smoothing model) with data from a new domain. To achieve this, it uses a pre-trained smooth model as the teacher model and proposes a simple loss function to make the student model also learn a similar smoothness. With a good teacher model, the student model can be trained very quickly, much faster than training from scratch.

Although some reviewers and I found the novelty of the work is limited, all of us agree that this work has empirical values in how to efficiently train a robust neural network in practical scenarios. Therefore, I recommend acceptance.
confidence: Less certain
Decision: Accept