Improving Generalization and Robustness in SNNs Through Signed Rate Encoding and Sparse Encoding Attacks

Authors: Bhaskar Mukhoty, Hilal AlQuabeh, Bin Gu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Spiking Neural Network, Adversarial Examples, Adversarial Robustness, Rate Encoding

Abstract:
Rate-encoded spiking neural networks (SNNs) are known to offer superior adversarial robustness compared to direct-encoded SNNs but have relatively poor generalization on clean input. While the latter offers good generalization on clean input it suffers poor adversarial robustness under standard training. A key reason for this difference is the input noise introduced by the rate encoding, which encodes a pixel intensity with $T$ independent Bernoulli samples. To improve the generalization of rate-encoded SNNs, we propose the *signed rate encoding* (sRATE) that allows mean centering of the input and helps reduce the randomness introduced by the encoding, resulting in improved clean accuracy. In contrast to rate encoding, where input restricted to $[0,1]^d$ is encoded in $\\{0,1\\}^{d\times T}$, the signed rate encoding allows input in $[-1,1]^d$ to be encoded with spikes in $\\{-1,0,1\\}^{d\times T}$, where positive (negative) inputs are encoded with positive (negative) spikes. We further construct efficient \textit{Sparse Encoding Attack} (SEA) on standard and signed rate encoded input, which performs $l_0$-norm restricted adversarial attack in the discrete encoding space. We prove the theoretical optimality of the attack under the first-order approximation of the loss and compare it empirically with the existing attacks on the input space. Adversarial training performed with SEA, under signed rate encoding, offers superior adversarial robustness to the existing attacks and itself. Experiments conducted on standard datasets show the effectiveness of sign rate encoding in improving accuracy across all settings including adversarial robustness. The code is available at https://github.com/BhaskarMukhoty/SignedRateEncoding

Decision: accept (poster)
Reviewer scores: 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1729156124585

thread_type:
official_review

<thread_object>
creator: reviewer_rmy3

<content>
summary:
This paper proposes a new encoding method for SNN input, called Signed Rate Encoding (sRATE), which is experimentally shown to reduce the randomness of encoded inputs through statistical analysis. Additionally, the paper introduces a Sparse Encoding Attack (SEA).

strengths:
S1. The sRATE is newly proposed and the analysis from randomness reduction is good.

S2. Authors conduct extensive experiments in evaluating the difference between RATE and sRATE SNNs, including adversarial training with various attacks.

weaknesses:
W1. The theoretical analysis needs improvement. From the current analysis, it is unclear why sRATE (Eq. 12) reduces randomness compared to RATE (Eq. 9). Both x and $|x-\mu|$ (I believe the $x$ in Eq. 12 is actually $x-\mu$, though this is not explicitly stated in Section 3.2) belongs to [0,1], meaning that the actual ranges of $k1$ and $k2$ are the same. It is the experimental results (Table 1) that demonstrate the advantages of sRATE, which weakens the motivation behind the sRATE design.

W2. The novelty of the proposed SEA method seems limited. If I understand correctly, the solution in Eq. (16) appears to be a straightforward application of the FGSM attack to RATE/sRATE encoded inputs. The idea of selecting $k$ elements based on the top $k$ gradients in the SEA method is fairly straightforward. I am also somewhat confused by the results—since the perturbations generated by SEA can be considered a subset of those generated by FGSM (with SEA essentially removing some of the perturbations from the FGSM attack), it is unclear why SEA outperforms FGSM, as shown in Table 5.

W3. The writing could be improved. In my view, Lemma 1 and Theorem 2 present concepts that are fairly straightforward, and it may not be necessary to dedicate much space to them in the main text. Other writing-related comments (which do not affect my overall rating) are summarized in the Questions section.

questions:
My primary concerns are outlined in the Weakness section. The following points are related to writing improvements and do not affect my overall rating.

Q1. As mentioned in W1, I believe the $x$ in Section 3.2 should actually be $x-\mu$, since $x$ could otherwise be negative. However, the authors did not clarify this point.

Q2. It seems unnecessary to define 'the positive and negative part' of a real number (Lines 197-199). For the sake of simplicity and readability, I believe it would be clearer to directly use piecewise linear functions instead, as this would make the formulation easier to follow.

Q3. The use of the same notation $k$ for different concepts is somewhat confusing. For instance, $k$ is used to represent randomness in Section 3, but it also denotes the sparsity of perturbations in Section 4.2 and the experiments.

Q4. The legend in Figure 4 is unclear, as it is difficult to distingusih between the dashed and solid lines in the legend.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730360187594

thread_type:
official_review

<thread_object>
creator: reviewer_iqc1

<content>
summary:
The authors propose a signed rate encoding (SRATE) method that enables mean-centering of the input, reducing the randomness introduced by traditional rate encoding. This approach improves clean accuracy and enhances the generalization of rate-encoded spiking neural networks (SNNs).

strengths:
The paper is well-structured and easy to follow.
The performance results show improvement with the proposed method.

weaknesses:
Some details require further clarification.

questions:
In the method section, is the encoding applied only to convert images to spike inputs, or does it also occur within spiking neurons in the network?

How could this technique be adapted for use with event-based datasets?

In Section 4.1, why does the attack rely on a first-order approximation rather than directly using the loss? What would happen if the direct loss were used instead?

In Figure 4 and Table 5, there are inconsistent notations (e.g., lowercase "fgsm" and uppercase "FGSM," similarly for "pgd" and "PGD"). Do these represent different methods, or is this just a formatting inconsistency?

How are FGSM and PGD applied to rate-based models in this study?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730464476818

thread_type:
official_review

<thread_object>
creator: reviewer_2rdr

<content>
summary:
This paper aims to explore an input encoding that balances the robustness and generalization on clean input of the model. The authors propose the signed rate encoding (SRATE) method that improves the accuracy of the model. In addition, they also introduce the sparse encoding attack on the SRATE input. Through theoretical analysis and experiments on datasets like CIFAR-10 and CIFAR-100, the authors demonstrate that SRATE improves generalization, while adversarial training with SEA offers superior robustness compared to traditional methods.

strengths:
The proposed method utilizes the rate-coding input, which is not only more bio-plausible but also provides inherent advantages in robustness and sparsity, distinguishing it from models that rely on constant input encoding methods.

The authors provide a solid theoretical basis for the SRATE, demonstrating that it preserves the essential characteristics of Poisson input encoding while effectively reducing randomness. 

The authors proposed a novel attack method, offering a solution for finding optimal adversarial examples under binary and sparsity constraints.

weaknesses:
One drawback of this method is its relatively low accuracy on clean inputs, with even the non-adversarially trained model achieving only ~55% accuracy on the CIFAR-100 dataset. The significant drop in clean accuracy may not justify the robustness gains, which raises the questions about the effectiveness of this approach, particularly when other methods (e.g. adversarial training) may offer a better balance between performance and robustness.

questions:
Sparsity is a critical feature in neural systems, offering benefits in terms of computational efficiency and reduced energy consumption. Given that this model is based on rate encoding, I wonder whether the model with SRATE encoding also leverages these advantages? 

It would be valuable to know if the authors considered using the Expectation Over Transformations (EOT) method for adversarial attacks, especially since EOT is an effective approach to deal with models that incorporate randomness.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a signed rate encoding (SRATE) method that enables mean-centering of the input, reducing the randomness introduced by traditional rate encoding. The reviewers acknowledged a solid theoretical basis for the SRATE as well as extensive experiments in evaluation. One reviewer pointed out the lack of proof, but the main problem was actually about some clarifications in the current proof, which was raised by another reviewer and then addressed by the author in the rebuttal. Thus, I recommended an acceptance.