Clustering Effect of Adversarial Robust Models

Abstract: Adversarial robustness has received increasing attention along with the study of adversarial examples. So far, existing works show that robust models not only obtain robustness against various adversarial attacks but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear. In this paper, we interpret adversarial robustness from the perspective of linear components, and find that there exist some statistical properties for comprehensively robust models. Specifically, robust models show obvious hierarchical clustering effect on their linearized sub-networks, when removing or replacing all non-linear components (e.g., batch normalization, maximum pooling, or activation layers). Based on these observations, we propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy.


Reviews and author responses:

 Thanks for your valuable suggestions.

**Q1.** In Eq. 9, how did you compute the features $z_i$?
 
**A1.** Thanks for your detailed reading. They are directly extracted from the feature before the FC layer.

**Q2.** Does the improvement in robustness come from the regularization term itself or the class hierarchy?

**A2.** Thanks for the valuable suggestions. We compare our method with the suggested ClusTR (Reference [1]). The results are as follows.

| AT ResNet-18s (CIFAR-10) | Ours    | ClusTR  |
| ------------------------ | ------- | ------- |
| ACC against AutoAttack   | 43.21\% | 0.03\%  |
| ACC against PGD-20       | 47.78\% | 35.04\% |

In addition, we analyze more on our method when designing a similar regularization loss yet with the wrong class hierarchy penalty(Ours\_w). Specifically we add hierarchical clustering penalty between fine classes (0,2,8,9) v.s. (1,3,4,5,6,7) for CIFAR-10, which should be (0,1,8,9) v.s. (2,3,4,5,6,7). The results are as follows. 

| AT ResNet-18s (CIFAR-10) | Baseline AT | Ours    | Ours_w  |
| ------------------------ | ----------- | ------- | ------- |
| ACC against PGD-20       | 46.50\%     | 47.78\% | 45.36\% |

The two experiments show that such improvement mainly comes from the clustering penalty in the class hierarchy.

**Q3.** The observations are based on the CIFAR-10 dataset.

**A3.** We also analyze them on STL and TinyImageNet. The correlation weight matrices of STL and TinyImageNet are shown in Figure 3. The domain adaptation performance of TinyImageNet is also shown in Figure 5 and the numerical results of TinyImageNet are shown in Table 6 in the Appendix. 

**Q4.** The '+C' operation has a limited effect on the robustness improvement.

**A4.** Yes. In fact, we did not aim to improve adversarial robustness as a new defense method. Our method provided a nontrivial understanding and a general improvement for adversarial robustness. Then we further explore and focus on some downstream tasks correlated with robust models. 

In addition, we also add the results of TRADES here.

| ResNet-18s (CIFAR-10) | TRADES  | TRADES+C(Ours) |
| --------------------- | ------- | -------------- |
| ACC against PGD-20    | 52.80\% | 53.71\%        |

 Thanks for your nice suggestions and detailed reading.

**Q1.**  In Algorithm 2, why retrain $\mathcal{F}$ from scratch? Would the fine-tuning work?

**A1.** Firstly, in our experimental settings, we assume that we do not know any category information of the dataset before training the network. So we need to pre-train the network to get the weight correlation matrix $C$ and $C_\mathrm{op}$. Secondly, fine-tuning works as well.

**Q2.** Is it hierarchical classification enhanced by robustness?

**A2.** Yes, hierarchical classification is enhanced by robustness. As shown in Figure 4, we compared the attacking confusion matrix of the robust model and the non-robust model, and we can see that the robust model shows the obvious class hierarchy in attack success rate (ASR). This phenomenon shows that hierarchical classification in the robust model makes it more difficult to attack across superclasses.

**Q3.** Figures 5 and 6 need a clearer description.

**A3.** Thanks for your valuable suggestions. We will carefully re-draw the Figures 5 and 6. In addition, for a better vision and understanding, we list a table of the shapes regarding to some model settings. In Figures 5 and 6, we try to show that robust(R) models as well as clustering effect enhanced robust(R+C) models show overall better performances than non-robust(NR) models in domain adaption tasks, even under all settings ($\textit{i.e.},$ different models, data sets or robustness levels).

| Models                         | NR       | R       | R+C(Ours) |
| ------------------------------ | -------- | ------- | --------- |
| Before fine-tuning(left panel) | Circle   | Square  | Star      |
| After fine-tuning(right panel) | Pentagon | Diamond | Octagon   |

 Thanks for your detailed reading and comments.

**Q1.** The description in line \#158 does not match the corresponding Figure 3(b).

**A1.** Thanks for your detailed comments. That is a typo. Models trained on STL-10 are shown in Figures 3(a)(e), where fine labels (0,2,8,9) belong to 'Non-animal' as mentioned in line \#158. Figures 3(b)(f) refer to models trained on CIFAR-10, where fine labels (0,1,8,9) belong to 'Non-animal'.

**Q2.** Discuss the operation of extracting the linear sub-network from the original DNN.

**A2.** Overall, we try to show a connection between the linearized weight space with the feature space. Firstly, we can simply start from a linear model, where $f(x)=k \times x$. Then the connection directly holds for weight vectors and feature space. And figures confirm that robust models show better clustering effect (see the weight matrix figures in https://www.dropbox.com/sh/yljxu4rnj9j47xc/AACZDUk1dq4mibx2Bi-Ucnf9a?dl=0). Secondly, as for the non-linear DNNs, the proposed weight extraction method is universal on the whole dataset. The extracted weight vectors represent for some linearized class-wise weight directions.We then compare the correlation weight matrix $C$ of the linearized sub-networks with the feature distance matrix $E$ of the original networks. More feature distance matrix figures are shown in the following link (https://www.dropbox.com/sh/yljxu4rnj9j47xc/AACZDUk1dq4mibx2Bi-Ucnf9a?dl=0). The figures reveal a nontrivial connection with matrices $C$ and $E$. They show the similar clustering effect. So the connection holds from linear sub-networks to original DNNs.

**Q3.** The '+C' experiment with TRADES.

**A3.** In fact, our proposed method did not aim to be a new adversarial robustness improvement technique, but focused more on the nontrivial understanding of adversarial robustness and the further benefit in some downstream tasks ($\textit{e.g.},$ domain adaption). Therefore, we did not add many defense baselines. However, as you suggested, we add extra experiments with TRADES, and the results are as follows.

| ResNet-18s (CIFAR-10) | TRADES  | TRADES+C(Ours) |
| --------------------- | ------- | -------------- |
| ACC against PGD-20    | 52.80\% | 53.71\%        |

**Q4.** Can $C_\mathrm{op}$ be effective on large-scale complex hierarchical datasets?

**A4.** Yes. We believe it works on large-scale complex hierarchical datasets. In fact, we optimize $C_\mathrm{op}$ by adjusting some thresholds to get the class hierarchy from the block clustering effect. So the experiments on a large-scale complex hierarchical dataset still work well.



This paper explores adversarial robustness via focusing on linear sub-networks. Through back propagation on linear sub-networks of a pre-trained network, an implicit weight expression which represents approximated class-wise responses can be obtained. With this matrix, the correlation among classes can be computed for exploring the hierarchy of classes. Similarly, the feature distance for different classes aligns well with class hierarchy through empirical observations. Based on these observations, the authors propose to impose feature distance regularization based on the computed class hierarchy.  Pros:

* This paper is well-organized and easy to follow.
* The motivation seems reasonable and natural from sufficient observations. 
* Extensive experiments are conducted on various tasks and datasets. The comparison with other baselines show superiority. 

Cons:

* There exist some confusing descriptions. For example, the authors mention that labels (0,2,8,9) belongs to ‘Non-animal’ in Line 158, however, it seems labels (0,1,8,9) are actually highly- correlated in Figure 3(b).

* The authors propose to extract a linear sub-network from a DNN through removing all the non-linear operations, and introduce a simple algorithm to obtain the weight expression of DNNs. However, the influence of this approximation is not discussed since most DNNs contains non-linear operations, such as ResNet-18 evaluated in experiments.

* Lack of a large number of defense baselines, such TRADES, etc. in Table 1.

* The approximate matrix C_{op} are binary. What if there exist a complicate class hierarchy in larger datasets, such as ImageNet?

 No potential negative societal impact.

The paper proposed a novel method of linear model exploration by extracting their implicit linear matrix expression and find the weight clustering effect. The paper also claimed that the adversarial robustness shows close relationship with such clustering effect, which enables the further understanding and improvement of various tasks in the experimental parts. Overall, I like the analysis in the paper, it provides an interesting and reasonable way to understand adversarial robustness via linearity of weights.  Strength:
- The goal of the paper is to contribute to an explanation of the underlying mechanism of adversarial robustness in DNN, and to exploit this explanation to improve both robustness and domain adaptation. Thus, the subject of paper is clearly of interest for the adversarial community.
- The general approach of the paper is to build two summary descriptors of the linear component of a DNN in the form of square matrices, each entry being associated with a pair of classes (Section 3). In the next section, these descriptors are exploited to create a regularizer for ordinary classification, and for domain adaptation tasks. The approach presented here is novel and convincing with a detailed explanation. 
- Section 4 provides a large number of good quality experiments. The penalty used for regularization is based on an inferred hierarchical clustering, itself build through an approximation of the “correlation” matrix. These experiments show moderate improvement. Other than adversarial robustness, the authors also applied the understanding of weight clustering on domain adaption study, showing the method has a general usage. 

Weakness:
- In Algorithm 2, why do authors need to retrain F from scratch? Would the fine-tuning with regularization loss work?
- The experiments on attacking confusion matrix is somehow confusing. Is the hierarchical classification enhanced or not? And is it enhanced by robustness, or is robustness enhanced by the use of the hierarchical classification penalty?
- The explanations for figures 5 and 6 should be more clear. Currently, it is not so easy to follow when reading the figures.  see the above weakness.

This paper analyzes the statistical regularity of robust models and non-robust models on their linear sub-networks and show that the robust models tend to show clustering effect aligned with the class hierarchy. A plugged-in hierarchical clustering training strategy is proposed to enhance adversarial robustness.  The observations and findings are interesting, which gives me a lot of insights.

My main concerns:
1. In Eq. 9, how did you compute the features z_i?
2. [1][2] show that regularization terms can benefit AT. It is not clear your robustness improvement comes from the regularization term itself or the class hierarchy. Can you conduct more ablation studies to explain this issue?

 Limitations:
1. The observations are based on CIFAR-10 dataset. The class hierarchy might not be clear on other datasets.
2. The robustness improvement is limited.

Meta-review: The paper targets the problem of how to understand the mechanism of adversarial robustness, which is a hot research topic. The authors presented that adversarial robust models have a clustering effect. Specifically, robust models show a clear hierarchical clustering effect on their linear sub-networks. By exploiting the effect, the authors illustrated how to further boost adversarial robustness. We think the paper will interest many readers.

Decision: accept