Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper suggests using the activation function k-Winners-Take-All (k-WTA) in deep neural networks to enhance the performance of adversarial defense. Their experiments show that the robustness is improved when they simply change the activation function to k-WTA. They also give reasonable theoretical analysis for their approach. 

I find that the idea is simple and elegant. Since they only change the activation function, their approach can be easily applied to almost all network structures and training processes. Their experiments show that the robust accuracies are significantly improved on all evaluated methods when they use the k-WTA activation function. I also appreciate their theoretical analysis. They show that k-WTA makes the network very discontinuous with respect to the input x, and thus the adversary could not get useful gradient information. In contrast, if the network is wide enough, then the discontinuities with respect to the weights w is sparse. This is why the network is still trainable though itself is not continuous.

The paper is also well-written and easy to follow. I recommend the acceptance of the paper.

One limitation of this paper is that their approach mainly focuses on defending gradient based attack. But I agree that the gradient based attack is currently almost the best attack method.

A minor question:
- How do we choose k in general? What is the behaviour for different k?

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The authors propose using k-winner take all (k-WTA) activation functions to prevent white box adversarial attacks. A k-WTA activation functions outputs the k highest activations in a layer while setting all other activations to zero. The reasoning given by the authors is that k-WTA activation functions have many discontinuities with respect to the input space. This makes it more difficult for attacks to use gradient information. The authors note that networks with k-WTA activation functions are still easy to train because, for a given input, the sub-network that is activated becomes more stable as training progresses. Therefore, it is not as discontinuous in the parameter space.

The authors test their method with 5 different adversarial attacks and train with 4 different training methods. They use the CIFAR10 and SVNH datasets.

The experiments showed that using k-WTA activation functions resulted in consistent improvements over ReLU activation functions in model robustness to white-box adversarial attacks when training with and without adversarial training methods. While, in the worst case, ReLU networks were around 50%-58% accurate in the face of adversarial attacks, k-WTA has accuracy that is usually 5%-17% higher.

While the novelty of this paper is low, the switch from ReLU to k-WTA appear relatively simple and yields better results than that of ReLU.

Other comments:
I don't think that this claim can be made without experimental evidence and should be removed:
"We are not aware of any possible smooth approximation of a k-WTA network to launch BPDA attacks.
Even if hypothetically there exists a smooth approximation of k-WTA activation, that approximation
has to be applied to every layer. Then the network would accumulate the approximation error at each
layer rapidly so that any gradient-estimation-based attack (such as BPDA) will be defeated."

Question:
I see the \gamma parameter of k-WTA is updated with a certain schedule that includes some finetuning. Including this finetuning, is the final k-WTA network trained with the same number of iterations as the ReLU network? Are all the other hyperparameters the same?

** After Author Response **
Changing from weak accept to accept

The authors have addressed my concerns and I believe the paper can provide significant value to those interested in adversarial robustness.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper addresses the important question of improving the robustness of deep neural networks against adversarial attacks. The authors propose a surprisingly simple measure to improve adversarial robustness, namely replacing typical activation functions such as ReLU with a k-winners-take-all (k-WTA) functions, whereby the k largest values of the input vector are copied, and all other elements of the output vector are set to zero. Since the size of input and output maps varies drastically within networks, the authors instead use a sparsity ratio \gamma that calculates k as a function of input size. k-WTA networks can be trained without special treatment, but for low \gamma values the authors propose a training schedule, whereby \gamma is slowly reduced, then re-training takes place, until the desired value of \gamma is reached. The presented effect is backed up by extensive theoretical investigations that relate the increased robustness to the dense introduction of discontinuities, which makes gradient-based adversarial attacks harder. A small change in an input signal can change the identity of the "winning" inputs, and thus in a sub-sequent matrix multiplication make use of other rows or columns, thus allowing arbitrarily large effects due to small input variations. Empirical evaluations in CIFAR and SVHN for a variety of attacks and defense mechanisms demonstrate the desired effects, and illustrate the loss landscapes due to using k-WTA.

I think the paper is a valuable and novel contribution to an important topic, and is definitely suitable for publication at ICLR. In principle there is just one novel idea, namely using k-WTA activations to improve adversarial robustness, but this claim is investigated thoroughly, in theory, and demonstrated convincingly in experiments. The paper is well written and tries to address all potential questions one might have surrounding the basic idea. There is code available, and the idea should be simple to implement in practice, so I would expect this paper to have large impact on the study of adversarial robustness.

I appreciate the thorough proofs of the claims in section C of the appendix, but I did not review all proofs in  detail. 

Potential weaknesses that should be addressed:
1. Section 4: I would propose to fully define A_rob here or at least provide a reference.
2. From Table 1 it seems that using k-WTA leads to a quite noticeable drop in standard accuracy, especially for sparse \gamma, which leads to the best adversarial robustness. Can you please comment on whether the full ReLU accuracy in the natural case can always be recovered by k-WTA networks, e.g. with larger \gamma?
3. Since small changes have a big effect in k-WTA, it should be investigated how robust the k-WTA networks are with respect to more natural perturbations, e.g. noisy input, blurring, translations, rotations, occlusions, etc. as introduced in (Hendrycks and Dietterich, 2019). It would be critical if such perturbations have a stronger effect on k-WTA.
4. Is there any intuition about whether k-WTA should be used everywhere in a deep network, or whether it makes sense to mix k-WTA and ReLU functions? 

Minor comments:
5. WTA networks are very popular in computational neuroscience and are even hypothesized to represent canonical microcircuit functions (see e.g. Douglas, Martin, Whitteridge, 1989, Neural Computation, and many follow-up articles). You cite the work of Maass, 2000a,b already, it could be interesting to link your work to other papers in that field who motivate WTA from a biological perspective.
review_title: Official Blind Review #1

Meta-review: This paper presents new non-linearity function which specially affects regions of the model which are densely valued. The non-linearity is simple: it retains only top-k highest units from the input, while truncating the rest to zero. This also makes the models more robust to adversarial defense which depend on the gradients. The non-linearity function is shown to have better adversarial robustness on CIFAR-10 and SVHN datasets. The paper also presents theoretical analysis for why the non-linearity is a good function.

The authors have already incorporated major suggestions by the reviewers and the paper can make significant impact on the community. Thus, I recommend its acceptance.
Decision: Accept (Spotlight)