Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes an ANN training method for improving adversarial robustness and generalization, inspired by biological sleep.

I'm leaning towards accepting as it seems to be an original concept and has fairly extensive empirical results that are somewhat promising.

The idea of a sleep phase as an alternative to explicit adversarial or generalization training is interesting. The results suggest that the approach works reasonably well in many cases.

Suggestions for improvement / clarification:
- The mapping from biological sleep to the actual algorithm + pseudocode used could benefit from more thorough explanation. It is not clear which choices are arbitrary vs well-principled.
- Was the optimal sleep duration determined empirically for each experiment?
- I agree with the authors' proposed future work of better understanding and standardizing this approach.
- Consider combining this approach with the existing adversarial or generalizing approaches (instead of as an alternative). Do they complement each other?
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Disclosure on reviewer's experience: I am not an expert on adversarial attack methods or defenses, but I am well read in the general literature on robustness and uncertainty in deep neural networks.

The authors present a biologically inspired sleep algorithm for artificial neural networks (ANNs) that aims to improve their generalization and robustness in the face of noisy or malicious inputs. They hypothesize that "sleep" could aid in generalization by decorrelating noisy hidden states and reducing the overall impact of imperceptible perturbations of the input space. The proposed sleep algorithm broadly involves 1) converting the trained ANN to a "spike" neural network (SNN), 2) converting the input signal (pixels) to a Poisson distributed "spike train" where brighter pixels have higher firing rates than darker pixels, 3) propagating the neuronal spikes through the SNN, updating weights based on a simplified version of spike-timing-dependent plasticity (STDP), and 4) converting the network back to an ANN after the sleep phase has finished. They present a detailed comparative study spanning three datasets, four types of adversarial attacks and distortions, and two other baseline defense mechanisms, in which they demonstrate significant improvements (in some cases) of the sleep algorithm over the baselines.

The core concept behind the authors' work is novel and interesting, and the experimental design is thorough and well controlled. Although the results are (I would argue) somewhat mixed, they are nonetheless positive enough to encourage more work in applying "sleep" and other relevant ideas from neuroscience to the problem of robustness in deep neural networks. I have some questions and concerns which I will detail per-section below, but overall, I believe that this paper is a valuable contribution to the literature and should be accepted once the authors have made a few necessary revisions.

Section 1: Introduction

"We report positive results for four types of adversarial attacks tested on three different datasets (MNIST, CUB200, and a toy dataset) ..."

It's debatable whether or not the results from the CUB-200 dataset are positive. The sleep algorithm fails to outperform the baselines for each attack type (except for an almost negligible advantage in accuracy on JSMA) and barely even outperforms the control network in most cases (2/4 attacks it actually underperforms the control). I think the authors should consider rephrasing this statement to better reflect the actual results.

Section 2: Adversarial Attacks and Distortions

FGSM: The notation used here is somewhat inconsistent with the source paper. Goodfellow et al use epsilon to denote what I think the authors call eta, and call the second term, epsilon*sign(grad(J)), eta. Furthermore, the authors state that "this represents the direction to change each pixel in the original input in order to decrease the loss function." But this doesn't make sense. An adversary should want to *increase* the loss function enough to cause a misclassification. Goodfellow et al use this expression to formulate a L1-like regularization term and describe the training procedure "minimizing the worst case error when the data is perturbed by an adversary", which seems more sensible. This section should be rewritten to be more consistent with the source.

Section 3: Adversarial defenses

Regarding distillation: "We use T=50 to compare with the sleep algorithm"

The authors should elaborate a bit more on the reasoning for this choice. It seems very arbitrary.

Sectioin 4: Sleep algorithm

1. Algorithm 1: Why is line 9 inside of the for loop? It doesn't seem to be at all dependent on t. One would expect the input to only need to be converted once. Additionally, in lines 11-13, the l's in W(l,l-1) and similar should be unbolded. It's confusing that the format changes (unless I am missing something and it's actually a different variable).

2. Spike trains should be more rigorously defined, preferably with formalized notation. It's a bit unclear exactly what they are from the current text. Are they just parameters for a Poisson? Or outputs from a poison over T time steps? Or something else?

3. "weights are scaled by a parameter to induce high firing rates in later layers"
It would be good to include more details on this parameter, how the values are chosen, and the intuition behind this idea. I assume it's because of higher level feature representations in later layers of deep neural networks.

Section 5: Results

1. It's confusing that sometimes accuracy refers to classification accuracy and sometimes adversarial attack accuracy. I would recommend assigning a different name to the latter, or making sure that a qualifier precedes every reference to "accuracy" in this section.

2. In the second section of the results table (which is missing a label), why is the JSMA value for Defensive Distillation bolded? The distance measures for both the control network and for fine-tuning are higher. It seems like fine-tuning should be the one bolded.

3. Figure 1: caption is incorrect; it states "adversarial attack accuracy" and it should be "classification accuracy", otherwise the plots make no sense.

4. "we observe that in the Patches and CUB-200 dataset, sleep has beneficial results in moving the accuracy function above the other defense methods"
It should be noted that this is only true for eta < 0.1. After that, sleep and the control both converge to 50% accuracy. Also this sentence should be reworded to be less visual and more quantitative (e.g. sleep tends to have higher median accuracy scores than the other methods for eta < 0.1).

5. "We observe that performance continued to drop after a sufficiently large amount of noise was added"
More than that, the other methods converged to a small band of accuracy values; sleep continued to deteriorate. This is a significant difference. It would be a good idea to re-run this experiment with a binary classification problem (e.g. only two digits of MNIST) and see if this phenomenon still occurs. Then, the noisy sleep classifier predictions could simply be inverted to get improved accuracy scores.

6. In the analysis of JSMA, as noted before,, it's rather dubious to claim that sleep had any kind of significant effect on the attack success rate (or distance) for CUB-200. I would rewrite this section to better represent the results.

7. Figure 2 formatting: Legend is overflowing out of the first figure. Additionally, the legend colors should be made to match across all three figures, and the legend should either appear in all three (if necessary for some reason) or only in one.

8. Figure 2: The caption is incomplete and possibly incorrect. It's not clear why the first and last figures differ from each other, and the caption does not indicate this. The caption also only mentions two datasets, even though it says "for the following three datasets".

Appendix:

General formatting needs improvement. A lot of figures are off-centered, text misaligned, missing axis labels, etc.
review_title: Official Blind Review #1

Meta-review: "Sleep" is introduced as a way of increasing robustness in neural network training. To sleep, the network is converted into a spiking network and goes through phases of more and less intense activation. The results are quite good when it comes to defending against adversarial examples. Reviewers agree that the method is novel and interesting. Authors responded to the reviewers' questions (one of the reviewers had a quite extensive set of questions) satisfactorily, and improved the paper significantly in the process. I think the paper should be accepted on the grounds of novelty and good results.
Decision: Accept (Poster)