Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new scheme for training the SNN model to achieve text classification in the NLP community. Due to the fact that it is hard to directly train SNN for language tasks using the error backpropagation through time. The authors start with an SNN converted from a normally-trained tailored network and perform backpropagation on the converted SNN. The pre-trained word embeddings are also transformed for SNN based text classification. Pre-trained word embeddings are projected into vectors with positive values after proper normalization and shifting, which can be used to initialize tailored networks and converted to spike trains as input of SNNs. 

If not considering the contributions of SNN for NLP tasks, the transformation from ANN to SNN is not a new idea actually. This paper is well-written, and the structure is clear for readers to follow.
strength_and_weaknesses: Strength 
1. design SNN for text classification further extends the applications of SNN model; 


Weaknesses
1. after a series of operations, the ANN is transformed into SNN version, however, the accuracy and running efficiency are all dropped? A natural question is do we have to conduct such a transformation? 
2. the Transformer networks achieve remarkable performance in the NLP tasks, would people select the SNN for their application? I doubt this selection. 
3. The experimental results are not impressive. 
clarity,_quality,_novelty_and_reproducibility: 1. the writing of this paper is good. It is clear for readers to follow and read. 
2. the novelty is OK, but I doubt the significance of the transformation from ANN to SNN model. 
3. Not sure about the reproducibility of this work. 
summary_of_the_review: This paper proposes a new scheme for training the SNN model to achieve text classification in the NLP community. Due to the fact that it is hard to directly train SNN for language tasks using the error backpropagation through time. The authors start with an SNN converted from a normally-trained tailored network and perform backpropagation on the converted SNN. The pre-trained word embeddings are also transformed for SNN based text classification. Pre-trained word embeddings are projected into vectors with positive values after proper normalization and shifting, which can be used to initialize tailored networks and converted to spike trains as input of SNNs. The writing of this paper is good. It is clear for readers to follow and read. but I doubt the significance of the transformation from ANN to SNN model.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on applying spiking neural networks (SNNs) to text classification tasks. 

Spiking neural networks more closely mimic biological neural networks, with the main difference from typical artificial neural network being a temporal quality: each neuron accumulates a membrane charge over time (across inferences) and only transmits information when its threshold is hit. When this event happens, the neuron is said to fire (or spike), it resets itself, and its signal travels to the connected neurons which in response will increase or decrease their charge.

The main contribution is to convert and fine-tune a regular TextCNN architecture into its SNN counterpart. In particular,
(i) the given TextCNN is tailored to replace or remove unsupported operations such as word embeddings, negative values, biases, max-pooling.
(ii) This tailored NN is then trained with Gradient Descent
(iii) The corresponding SNN is initialized with the weights learned in the previous step, and then is fine-tuned using surrogate gradients.

Experiments are conducted on sentiment classification tasks.

strength_and_weaknesses: Strengths:
- Simple approach, easy to follow.

Weaknesses:
- Contributions are incremental and only marginally novel.
- This approach still requires training a regular TextCNN, partially defeating the motivation behind SNNs.
- Experimental section is quite limited and inconclusive. In particular, the lift from using fine-tuning vs simply using a converted SNN is rather marginal.
- Experimental robustness section: why are the "Clean" metrics in Table 2 different from the ones in Table 1. If that's because of the variance of results over different run, then the observed variance would invalidate the results and ranking presented in Table 1.
clarity,_quality,_novelty_and_reproducibility: The paper is quite easy to follow. However, being spiking neural networks quite a niche field, I would have hoped the authors had introduced them in more details to help the reader understand the SNN setup and differences from regular neural networks (if not in the main body, at least in the appendix).
summary_of_the_review: Given the limited novelty and improvements, my recommendation is to reject this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper applies SNNs to text dataset tasks. It proposes a method of converting a pre-trained standard ANN into an SNN. This improves on current methods by allowing the model to handle large vocabularies.
strength_and_weaknesses: 
(strengths)

Well-written and organized.

Extends viability of SNNs to large vocabularies.

A wide array of example tasks, in 2 languages.

(weaknesses)

The opening motivation, reducing energy consumption, is less impactful than it appears because the proposed method requires pre-training and fine-tuning a standard ANN before conversion to an SNN. So the only savings are over inference time. For a very few successful and widely deployed algorithms this will be significant, but for most algorithms during research, there is little difference in energy cost. This distinction should perhaps be clarified.

(Miscellaneous)

Table 1: is it possible to report mean +/- std dev over k-fold splits? This gives a much more powerful and informative report.

Table 2: Since the SNNs are still clearly harmed by the adversarial attacks, perhaps you could discuss whether there are ways to improve on this (or conversely comment that robustness, while much better than CNNs, is unlikely to improve).

Page 1: is the "question classification task" one of the 17 toy tasks, or is it substantially harder?

Page 1: "which is less biologically plausible": Does this matter at all? perhaps this clause could be removed.

Bottom page 2: Maybe replace "former" and "latter" with "conversion" and "spike-based" for clarity.

Section 3.1: "is trained by the same way as one would with " -> "is trained in the same way as"

Section 3.1: " some operations that cannot be realized by ": this raises an instant question in the reader "what operations?" which is not answered until 3.1.1. Perhaps add a pointer to that future list, eg "(listed in 3.1.1)"

3.1.1: "method is applied" -> method can be applied

Top of page 4: "embeddings learn" -> "embeddings to learn"

3.1.1 third bullet: It seems that removing biases would affect whether units activated or not. Why does removing biases not matter?

"clamp": maybe "clip" is more standard?

"dividing by sigma": divide by 6*sigma?

Figure 2: This should be moved to after eqn 2, so that the reader knows what the variables mean. In its current location, the figure is not comprehensible when encountered.

"illustrated in Figure 2" -> "2a"

3.2.3: It looks like the target vector increases as the vocabulary size. How big is it, and are there limits to vocabulary size due to this part of the architecture?

"3.2.3: "we list the efficient" -> "an"



clarity,_quality,_novelty_and_reproducibility: The paper is very well-written and organized, and is a compelling read.

The research program and experiments are well-scoped and well-thought out.

The findings extend the current art meaningfully.

The Introduction lays out relevant prior work. I do not have sufficient background in the literature to confirm completeness of citations or novelty (so the scores I give for Novelty might be wrong).
summary_of_the_review: The paper documents a well-designed research program and experiments that extend current art in the field of SNNs.

The paper itself is well-written and a pleasure to read.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a “conversion +fine-tuning” two-step method for training SNNs for text classification. Meanwhile, a new encoding method for converting pre-trained word embeddings to the spiking version is proposed. As results, the converted SNNs achieved comparable results compared with DNNs on text classification benchmarks. Also, the proposed model shows better robustness against adversarial attack than DNNs.
strength_and_weaknesses: Strength:
1. There are few works that demonstrate the efficacy of SNNs in language tasks. This paper makes a breakthrough.
2. The results are excellent. The proposed models achieve comparable results to the original TextCNNs.
3. The idea is clear and the paper is well organized.

Weakness:
1. The idea is not very novel. The whole pipeline is quite similar to the converting process of original SCNNs in the image recognition domain.
2. The spiking encoding may lead to the increasing of time steps during inference and training for each word embeddings that are converted to spike trains periodically. The authors did not mention that.
clarity,_quality,_novelty_and_reproducibility: The idea is clear and the paper is well written and organized. The idea is not very novel but the authors applied it in a new application. 
summary_of_the_review: This work fills the gap in the field of SNNs for text classification tasks. The experimental results are excellent and the conclusions are well supported. But the authors need to justify the impact of time steps during inference and training of the converted SNNs.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes and tested a "conversion + fine-tuning" method for spiking neural networks (SNN). They converted from TextCNN to SNN and demonstrated comparable performance as TextCNN. They also showed more adversarial robustness due to the conversion. Several reviewers objected that the results are no better than TextCNN and that one still needs to train the original TextCNN model. The authors were pretty clear that this was an inference-time procedure, and it is valid to argue that inference-time saving is the most important for a model (perhaps except for research). Since SNN is a new area, the AC and reviewers agree that the paper deserves the benefit of the doubt for proposing an effective conversion procedure.

In addition to some of the doubt raised, the benchmark and model are fairly simple and both the base and converted performances are comparable to bag of words . So the results are vulnerable to the objection that one does not need to do too good a job at the conversion. (The first citation  (Krizhevsky et al., 2017) also appears to have the wrong year. )




justification_for_why_not_higher_score: some doubts from the reviewers and AC
justification_for_why_not_lower_score: reviewer and AC concensus
Decision: Accept: poster