Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study aims at distilling feature representations (mainly the last convolution layer) into robust and non-robust features using information bottleneck. For every layer, the feature maps are partitioned into robust and non-robust for each individual example. After finding this, the paper first illustrates that selectively propagating robust features of adversarial examples built for a robust network will not degrade the accuracy but propagating non-robust features does which to some extent illustrates that the information bottleneck method has correctly identified the robust and non robust features. They also illustrate that the adversarial class label has high correlation to the Non-robust features.

main_review: The paper has some interesting insights but they all correspond to what is expected and it doesn’t provide much new understanding of robustness. Also, the paper's clarity can be significantly improved, for example, there are parts which are referenced before and then introduced after (for example, Z^a_{nr} is referenced in section 3.2 and figure 3 but is defined for the first time in section 3.3). After reading the paper with care, I am still not sure if I understood the experiments completely. The paper extends the idea of input data having robust and non-robust features to deeper “learned” representations of a Neural Net. While in the original adversarial examples are features not bugs paper, the robust and non-robust features co-existed in each channel, in this paper the authors classify each channel as robust or non-robust. I do wonder whether more straightforward methods exist for this partitioning purpose — possibly, simply using the ell_2-norm of the gradient of the loss w.r.t. the feature map —  which can result in similar qualitative results? 

One interesting factor seems to be from the fact that the non-robust deep features are also perceptually aligned. However, I am not sure if this is true in general, i.e., when we do not use regularizations used for visualizations and we partition deep features for naturally trained models. Do the authors have an intuition for that case? I do wonder if robust and non robust features of a naturally trained net are also perceptually aligned? The finding that the non-robust feature's visualizations for a natural example on an adversarially trained model visually look like the adversarial class is not shocking and is expected due to the perceptual alignment of robust networks where if you make an adversarial example with a large epsilon for them, the adversarial example looks like an image from the adversarial class. 

I do have the following other questions which could help me understand if I understood the paper good or not:

* Is there a reason for using a robust net in Table1? Does the network have to be adversarially trained to have clearly separable R and NR features?
* The NRF attack does not seem to be ell-infitiny bounded, is this correct?  If yes, comparisons of that with other attacks in table 3 does not seem to be fair.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work utilizes an Information Bottleneck (IB) to distill robust and non-robust features in neural networks (NNs). Experiments are performed to show that the distilled features are correlated with adversarial prediction and are human-perceptible. Finally, an attack mechanism intensifying the gradient of non-robust features is proposed.
main_review: Utilizing IB to capture relevant information is not a novel technique, but distilling robust and non-robust features in the intermediate layers of networks with it sounds interesting. The authors define the robustness and non-robustness of units by their correlation to the model prediction and separate informative features into robust and non-robust units by their tolerance of feature variation, which is a novel perspective to study the robustness of NNs. Experiments along with visualizations can well support their claims. There is also question: What is the connection between the robustness and informativeness? Does Figure 2 implies that non-robust features are less informative than robust features?
limitations_and_societal_impact: Please refer to the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to explicitly separate intermediate representation into robust and non-robust categories by information bottleneck, and demonstrate that the non-robust feature is related to adversarial prediction. An attack mechanism based on enforcing non-robust features gradient is proposed. The experimental results show the effectiveness of the proposed approach.

main_review: Information bottleneck(IB) has been widely used for feature disentanglement. It’s an interesting idea to connect IB to separate robust and non-robust features under adversarial settings.  Overall, the paper is well-organized. Hopefully, the authors can help address the following questions. 


**1.** In line 119, how did you calculate the inherent feature variation $\sigma_z$? Is it obtained by computing the variance of the same unit along the batch size dimension within a batch? I think it needs to be clarified since its maximal value directly determines the robust and non-robust index.


**2.** Amplifying the brittleness of non-robust features is proposed as an attack. It is a typical white-box and untargeted attack, of which the applicable scenario is limited these days. Usually, we evaluate the effectiveness of an attack by attacking a baseline defense approach. There are many defense works against adversarial attacks[1]. To make the paper stronger, I think the authors should try at least one defense strategy to verify the proposed attack.  


[1] [Towards Deep Learning Models Resistant to Adversarial Attacks.](https://arxiv.org/abs/1706.06083)  



**3.**  The channel index acts like a mask to filter out non-robust features, which is highly dependent on the dataset used. On the other hand, most of the experiments are done on CIFAR-10 and SVHN, which both are small datasets and nicely distributed. I wonder about the transferability to larger datasets of the proposed method

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper categorizes the intermediate features of a DNN into robust ones and non-robust ones. Through visualization, the non-robust ones display different semantic information from the true label, and thus potentially leads to the existence of AE.  The paper proposes a new AE generation algorithm targeting the non-robust feature, and beats the common attack baselines.
main_review: I like the overall idea of the paper. It delves deep into the potential cause of AE and shows an interesting perspective.

However, this paper suffers from the poor writing. There's no apparent grammar error, but the paper is almost un-readable because of wordy expressions, numerable inline definitions and lacking of concise algorithmic description. The algorithmic steps are buried in definition of terms and lengthy details. Here are some of my suggestions.

-- Present the definition of IB, the definition of robust and non-robust feature and the algorithm of distinguishing them in separate definition boxes or separate paragraphs. Never use inline definition for key terms.

-- Instead of immediately adding comments to the parameters in an equation, wait for the main content be defined and put analysis together. There's an analysis section, so the analysis should all go there... For each aspect of the analysis, add a paragraph header if subsection header takes too much space.

-- Make sure all terms in the key equations are defined. For example in Equation 3), the term f_{l+} is not defined. 

-- Use concise expression. For example, starting from Line 97 "Here, ..." to the end of that paragraph, the entire part can be replaced by "Since finding the exact value of the second term is computationally prohibitive, we use the heuristics in [18, 19]." Save the space for **your** contribution. 

-- Make the caption count. For example for the caption of Figure 3, explicitly say the non-robust feature displays semantics similar to the misclassified label of AE.

Overall, I like the contribution of this paper. However, in order for the community to better understand the paper and to more conveniently refer to the formulation in this work, I want to see a careful revision in writing. Therefore, I'm giving a score of 5 for now. 

============ Post Rebuttal Response =======================

Thanks for the response. It's unfortunate that Neurips doesn't allow uploading an updated draft. I will raise my score to 6 and trust the authors to make another pass over the writing thoroughly.
limitations_and_societal_impact: They are adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on robust and non-robust features in adversarial examples. This paper proposes a method of distilling robust and non-robust features via information bottleneck. Through analysis, this paper has shown the high correlation between the distilled features and the adversarial prediction. Based on that, the authors design a new attack that intensifies the gradient of non-robust features.
main_review: Pros:

1. This paper proposes a new way to distill robust and non-robust features via information bottleneck.
2. The detailed method are well illustrated and sounds feasible.
3. Extensive analysis on the distilled features is provided and the figures or tables are well presented.
4. The current experimental results have shown the strong performance of the proposed attack.

Cons:

1. The motivation for using the information bottleneck seems to be unclear. Why we need the information bottleneck? Why information bottleneck can help to distill robust and non-robust features precisely? It is better to clearly discuss more in the former part of this paper.
2. It is better to discuss more the difference between this paper and [1] since the latter also provides a way to deal with robust and non-robust features. What is the relationship between the definitions in the two papers of robust/non-robust features? What is the unique significance of using information bottleneck to distill robust/non-robust features? Since the latter paper has already demonstrated the high correlation between the prediction and robust/non-robust features and proposed an easy but effective way to figure out these features.

Minor Concerns/Questions:

1. Can the authors verify the effectiveness of the proposed attack method on a larger dataset? Can the authors discuss the impact of dataset size on the performance of the attack method?

[1]. Ilyas, Andrew, et al. "Adversarial Examples Are Not Bugs, They Are Features." Advances in Neural Information Processing Systems 32 (2019): 125-136.
limitations_and_societal_impact: It is encouraged to discuss more how to utilize the distilled robust features.
review_title: null

Meta-review: This paper utilizes the information bottleneck mechanism to disentangle robust and non-robust features in the representation space of a deep neural network. The paper shows a high correlation between the (wrong) prediction on an adversarial example and the identified non-robust features for the example. Finally, the paper proposes a new adversarial attack that aims to enlarge the gradient for the non-robust features to degrade the model performance.

Even though robust and non-robust features have been discussed in the context of adversarial examples before, this paper proposed an interesting and novel approach to disentangle these features in the representation space. The experiments in the paper do demonstrate a clear connection between the non-robust features and the performance of the model on adversarial examples. Furthermore, the identified adversarial attack is shown to be very effective in rendering the model useless. Thus, this paper will be a valuable addition to NeurIPS 2021.

There is some room for improvement in the writing of the paper. Please address the following issues in the revised version. Instead of formally introducing many notations, key definitions, and algorithmic details, they appear as part of various paragraphs. There is no discussion of the computational complexity issues, especially in the context of the proposed attack. Furthermore, how feasible is it to perform the proposed attack (NRF) as it requires first disentangling the robust vs. non-robust features and then accordingly perturbing those? Does NRF provide any guidance to train more robust models? Interestingly, it appears from Table 4, that model degradation is quite significant even when robust features are perturbed. In fact, perturbing the robust features is more effective than all baselines considered in Table 3 (obviously, except NRF). This point is glossed over in lines 284-288. Addressing these aforementioned issues will certainly improve the overall quality and impact of this paper. 
Decision: Accept (Poster)