MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification

Authors: Sajjad Amini, Mohammadreza Teymoorianfard, Shiqing Ma, Amir Houmansadr
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Adversarial Training, Sparsification, Robustness, Activation Functions, Proximal Operator

Abstract:
We present a simple yet effective method to improve the robustness of both Convolutional and attention-based Neural Networks against adversarial examples by post-processing an adversarially trained model. 
Our technique, MeanSparse,  cascades the activation functions of a trained model with novel operators that sparsify mean-centered feature vectors. 
This is equivalent to reducing feature variations around the mean, and we show that such reduced variations merely affect the model's utility, yet they strongly attenuate the adversarial perturbations and decrease the attacker's success rate.
Our experiments show that, when applied to the top models in the RobustBench leaderboard, MeanSparse achieves a new robustness record of $75.28$% (from $73.71$%), $44.78$% (from $42.67$%) and $62.12$% (from $59.56$%) on CIFAR-10, CIFAR-100 and ImageNet, respectively, in terms of AutoAttack accuracy. 
Code: https://anonymous.4open.science/r/MeanSparse-84B0/

Decision: reject
Reviewer scores: 5, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729277508089

thread_type:
official_review

<thread_object>
creator: reviewer_bquf

<content>
summary:
The paper introduces MeanSparse, a novel post-training method designed to enhance the robustness of convolutional and attention-based neural networks. MeanSparse works by sparsifying mean-centered feature vectors. This technique effectively decreases the success rate of adversarial attacks by minimizing the exploitable feature variations. The paper demonstrates that MEANSPARSE improves the robustness on datasets such as CIFAR-10, CIFAR-100, and ImageNet.

strengths:
1. The application of sparsification to mean-centered features is an innovative approach, as it targets non-robust features in a unique way.

2. Authors provide comprehensive results across multiple datasets and models

3. The paper is well-organized and clearly describes the underlying motivation, methodology, and results

weaknesses:
1. whether its effectiveness generalizes across different types of attention-based models, such as ViT?

2. Could you provide provide an analysis of how MEANSPARSE's effectiveness scales with model size? Authors only apply their method on large networks, like wrs-70. How about the performance of the method on smaller network, like ResNet-18, Swin-Small?

3. After reading the codes provided, I found that the MeadSquare will calculate the mean and var of the input data to get running_mean and running_var. I do no think the model should change its any parameters according to the test data.

4. The number of baselines in this paper is too few. Authors do not compare their methods with other sparsity methods.

questions:
1. Could the method be used for adversarial training and accumulate the running mean and running var during training? Then during testing, fix the running mean and var.

2. Could you provide a detailed analysis of how MEANSPARSE's effectiveness changes as the attack strength increases, like 16/255?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708461127

thread_type:
official_review

<thread_object>
creator: reviewer_ry3u

<content>
summary:
This paper seeks to improve adversarial training by inducing sparsity in the features of an adversarially-trained model. To achieve this, MeanSparse blocks variations in features within a given distance of the mean, which is intended to lessen the importance of non-robust features during training. Before describing their sparsification method, the authors provide intuition as to how non-robust features can be removed during training through regularizad optimization. Here, a regularization term is included in the training objective which penalizes the $\ell_0$ norm of mean-centered learned features. In theory, this would remove small deviations from the mean which don't result in significant reductions in loss. Inspired by this intuition, the authors then propose a sparsification operator, which is applied during forward propagation and explicitly blocks variation within a given distance of the feature mean. Severel implementation challenges are then addressed. Experimental results show that integrating MeanSparse into state of the art image classifiers can significantly improve robustness with negligible impact on clean performance.

strengths:
- The results show meaningful improvement in the robustness of SOTA adversarially trained models. Since this method can be applied post-training, it has the potential to lead to a significant jump across the board in standards for robustness to adversarial examples.
- The experimental section appears well designed. Results are presented for CIFAR-10, CIFAR-100, and Imagenet, and a representative variety of model architectures and adversarial attacks are tested.
- The method is also shown to be robust against adaptive attacks, as shown in Appendix A.4.
- I quite like the visualizations provided in Figure 1, they simply and effectively convey how MeanSparse operates.

weaknesses:
- The description of the MeanSparse technique is somewhat ambiguous to me. The term "feature" is often used, but never specifically defined. What features are being used here? Are you referring to input features? Activations of the final layer? Activations of some internal layer?
- I don't entirely agree with the provided intuition for blocking minor variations around the feature mean. Based off of the provided explanation, I would expect this approach to work when minor variations are blocked around high-probability feature values. However, it's not clear to me that the feature mean would always be a high-probability point.
- Appendix A.2 does provide results looking at $\ell_2$ bounded attacks, rather than $\ell_\infty$ bounded attacks. However, I think the topic of how MeanSparse performs against different threat models does warrant additional study. Threat models have been studied in which the adversary can introduce perturbations that are unbounded in $\ell_p$ space (i.e. [1]), and it is not obvious to me whether these types of attacks would have similar impacts on the distributions of features. If the claims made in this paper are limited to $\ell_p$ bounded attacks, I think that should be made explicit.

[1] Xiao, Chaowei, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. "Spatially transformed adversarial examples." arXiv preprint arXiv:1801.02612 (2018).

questions:
- Is the feature mean always a high-probability point? I would imagine that in certain multimodal distributions the mean is actually unlikely to occur. Might this occur in practice?
- One component of MeanSparse involves calculating the mean and standard deviations of features. Can you provide more information regarding what data is used to calculate these values? Specifically, are these statistics computed on benign or adversarially perturbed data? 
- How sensitive is the feature mean to class imbalances in the training set? If classes aren't evenly balanced (assuming feature distributions are different for different classes), could that result in changes in feature means?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724897796

thread_type:
official_review

<thread_object>
creator: reviewer_8mzy

<content>
summary:
The paper proposes a post-training method for enhancing the robustness of adversarially trained models. In particular, it inserts, at various layers, the MeanSparse modules which project the features (before activation) onto their mean computed over the training set if they are closer than a threshold to the mean itself. This has the goal of preventing an attacker from exploiting non-informative features to change the predicted class. In the experiments, several SOTA robust models are equipped with MeanSparse, which leaves clean performance unchanged while improving robustness.

strengths:
- The proposed method is efficient, can be applied to any pre-trained model without additional training, and incurs in limited additional inference cost.

- The experiments include several architectures, datasets and threat models.

weaknesses:
- The main concern is about the possible presence of gradient masking [A]. In fact, as mentioned in the paper, the MeanSparse operator induces zero gradients for the features which are projected. Since these are supposed to be the most common ones, and MeanSparse is applied in multiple points in the network, one can expect that the computed gradient might contain limited information, and thus the gradient-based attacks not work properly. While the paper tests black-box attacks, in this case the base model is (highly) robust, and the improvements given by MeanSparse are in the order of 1-3%, which might be of the same order or even smaller than the gap between white- and black-box (with standard query budget) attacks for the base model: then it is not clear that in this case this is sufficient to exclude gradient masking. A simple adaptive attack would consist in removing the projection operation when computing the gradient in the attacks, which would modify the backward pass of the model while preserving its predictions (a similar approach to BPDA [A]).

- The discussion in Sec. 3.1 seems a bit disconnected from the final approach (also, $z_{k-1}$ in Eq. (7) is not defined).

[A] https://arxiv.org/abs/1802.00420

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730740302563

thread_type:
official_review

<thread_object>
creator: reviewer_ik2x

<content>
summary:
This paper introduces a method called MEANSPARSE, which enhances the adversarial robustness of trained neural networks in a post-processing manner without compromising clean accuracy. The idea behind this method is to attenuate the reliance on non-robust features by modifying activation functions. Their experiments demonstrate a significant increase in the adversarial robustness of neural networks.

strengths:
1. The idea of manipulating activation functions to enhance adversarial robustness is quite interesting.
2. Extensive experiments have been conducted to support the proposed method.
3. The intuition behind the idea is provided, making it easier to follow.
4. The paper includes a discussion of its limitations.

weaknesses:
1. The explanation of the method in the Introduction is vague. On page 1, lines 37 and 44, you state: "In this work,..." twice, but the first sentence mentions "design of activation functions," while the second shifts to "sparsifying features to enhance robustness against adversarial attacks." These two sentences make it unclear what the key points of your work are and how investigating activation functions is related to sparsifying features. It would be better to provide a consistent expression for the main points and briefly describe the connection between them. I suggest a clear statement about your key contributions.

2. Lack of intuitive explanation of the proposed concept. On page 1, line 48, the proposed sparsity method "Mean-based Sparsification" is not well explained, and it’s followed only by a brief description of Figure 1. Without a clear explanation of the "sparsification operator", it is difficult to follow. I suggest adding a brief explanation of the "sparsification operation" you mentioned in line 90 and providing a concise mathematical definition or detailed description of how the sparsification operator works, perhaps with a simple example.

3. There are some writing errors in the paper. In Figure 2, "equation 3" appears at the end of the title, which is confusing. It would be better to remove it or rephrase the title.

questions:
1. On page 6, in the first and second paragraphs, you mention two approaches and choose the second one. I am curious why you selected the second approach as your method. Is it superior? Can you provide evidence to support your choice and a brief comparison of the two approaches, highlighting the advantages and disadvantages of each?

2. In Section 4 on page 7, I did not find the experimental settings. You only mentioned that the experiments were conducted using an NVIDIA A100 GPU. For better reproducibility, it would be helpful to include detailed experimental settings. I suggest providing details such as the software versions used, hyperparameters, data preprocessing steps, and any other relevant configuration details that would allow others to replicate their experiments
</content>
</thread_object>

</thread>
<thread>
creation date:
1732596785476

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions and Updates to Address Reviewer Feedback

comment:
We sincerely thank the reviewers for their insightful feedback and for recognizing the strengths of our paper. We have carefully addressed each comment and made several revisions to enhance the clarity and quality of the manuscript. Below is a summary of the changes made in the updated version:

$\textbf{Explanation of the MeanSparse Method in the Introduction:}$

To improve clarity, we revised the Introduction to include a concise explanation of the sparsification operator, supplemented by a simple example. This addition makes the introduction more accessible and easier to follow.

$\textbf{Figure 2 Caption:}$

We revised the caption of Figure 2 to enhance clarity and better describe the visualized content.

$\textbf{Comparison of Post- and During-Training Integration of MeanSparse:}$

In Section 3.3, we outlined two approaches to integrating the MeanSparse operator. We revised this section to include a detailed explanation of the differences between the two approaches, discussing the advantages and disadvantages of each.

$\textbf{Gradient Masking Concerns:}$

In response to reviewer feedback, we updated the Limitations section (section 4.4) to address potential concerns about gradient masking and adaptive attacks. We clarified that while MeanSparse improves robustness against non-adaptive attacks, its projection operation induces zero gradients, which could reduce efficacy against white-box adaptive attacks like BPDA. We also included results from preliminary experiments using a BPDA-like approach, demonstrating a decrease in robust accuracy when replacing the MeanSparse gradient with an identity function. These updates ensure a transparent and comprehensive discussion of MeanSparse's limitations.

$\textbf{Correction in Equation 7:}$

We corrected a typo in Equation 7, where $Z_{k−1}$​ was mistakenly written instead of $\bar a_{k−1}$. This has been addressed in the revised manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary** This work explores sparsity for adversarial robustness by proposing and evaluating a particular sparsity transform on deep representations. The MeanSparse operation is similar to (soft-)thresholding, in reducing differences about the center to zero, but distinct in maintaining values away from zero. In this case the center is set to the feature mean and a threshold hyperparameter $\alpha$ is scaled by the feature variance. This sparsity transform is applied either during adversarial training or after training and only during inference. The after training/post-training variation is favored and is implemented by calculating feature statistics over the training data without adversarial attacks. Evaluation on the standard AutoAttack benchmark shows improvement.

**Strengths**: altering activation functions is general and of interest (ik2X, ry3U) and computationally efficient (8mZY), the experiments are extensive w.r.t. architectures/datasets/threat models (ik2X, 8mZY, ry3U, bqUf), limitations are identified (ik2X), and the work is clear with an intuitive visualization and explanation help to understand the proposed operation (ik2X, ry3U, bqUf).

**Weaknesses**: the apparent robustness of the proposed operation could be explained away by gradient masking (8mZY), the explanation of the operation is vague (ik2X, ry3U), and experimental settings are not detailed (ik2X). Note that the weaknesses raised by bqUf have been discounted, although they are worthwhile points, because they were satisfactorily discussed in the response.

**Decision**: four expert reviewers choose marginal acceptance (ry3U: 6), marginal rejection (ik2X: 5), and clear rejection (8mZY: 3, bqUf: 3). Weaknesses concerning clarity or further results were addressed, and the vote for rejection by bqUf has been qualified accordingly, but the essential weakness about gradient masking and adaptive attacks (8mZY) remains. The meta-reviewer agrees with the reply by 8mZY to the rebuttal that "discussing the potential limitations without a quantitative evaluation is in this case is not sufficient, given that there exist a large body of work on adaptive attacks". The meta-reviewer therefore sides with rejection, but encourages the authors to revise their work with further study and attacks and resubmit, so that the evaluation of the proposed defense is more convincing to experts on adversarial robustness.