Learning Robust Kernel Ensembles with Kernel Average Pooling
Authors: Pouya Bashivan
Adam Ibrahim
Amirozhan Dehghani
Yifei Ren
Venue: Submitted to ICLR 2023 (2023)
Keywords: convolutional neural networks
topographical neural networks
adversarial robustness
ensemble models

Abstract: Model ensembles have long been used in machine learning to reduce the variance in individual model predictions, making them more robust to input perturbations. Pseudo-ensemble methods like dropout have also been commonly used in deep learning models to improve generalization. However, the application of these techniques to improve neural networks' robustness against input perturbations remains underexplored. We introduce \emph{Kernel Average Pool (KAP)}, a new neural network building block that applies the mean filter along the kernel dimension of the layer activation tensor. We show that ensembles of kernels with similar functionality naturally emerge in convolutional neural networks equipped with KAP and trained with backpropagation. Moreover, we show that when combined with activation noise, KAP models are remarkably robust against various forms of adversarial attacks. Empirical evaluations on CIFAR10, CIFAR100, TinyImagenet, and Imagenet datasets show substantial improvements in robustness against strong adversarial attacks such as AutoAttack that are on par with adversarially trained networks but are importantly obtained without training on any adversarial examples.

Under review as a conference paper at ICLR 2023
LEARNING ROBUST KERNEL ENSEMBLES WITH KER- NEL AVERAGE POOLING
Anonymous authors Paper under double-blind review
ABSTRACT
Model ensembles have long been used in machine learning to reduce the vari- ance in individual model predictions, making them more robust input perturba- tions. beeudo-ensemble methods like dropout have also been commonly used in deeplearning learning models to improve generalization. However, the application of these techniques to improve neural networks' robustness against input perturbations re- mains undcrexplored. We introduce Kernel Average Pool (KAP), a new neural network building block that applies the mean filter along the kernel dimension of the layer activation tensor. We show that ensembles of kernels with similar func- tionality naturally emerge in convolutional neural networks equipped with KAP and trained with backpropagation. Moreover, we show that when combined with activation noise, KAP models remarkably robust against various forms of ad- versarial attacks. Empirical evaluations on CIFAR 10, CIFAR (0, Tinylmagenet, and Imagenet datasets show substantial improvements in robustness against strong ad adversarial attacks such as AutoAttack that are on with ad adversarially trained networks but are importantly obtained without training on any adversarial exam- ples.
INTRODUCTION
Model ensembles have long been used to improve robustness in the presence of noise. Classic methods like bagging (Breiman 1996). boosting (Freund) [995] Freund al.] 1996), and random forests (Breiman 2001 are established approaches for reducing the variance in estimated prediction functions that build on the idea of constructing strong predictor models by combining many weaker ones. As result, performance of these ensemble models (especially random forests) is surprisingly robust to noise variables (i.e. features) (Hastic et al. 2009).
Model ensembling has also been applied in deep learning (hhou et al. 2001 Agarwal et al., 2021 Liu et al. 2021 Horváth et al.) 2022). However, the high computational cost of training multi- ple neural networks and averaging their outputs at test time often quickly becomes prohibitively expensive (also see work on averaging network weights across multiple fine-tuned versions (Worts- man et al. 2022). To tackle these challenges, alternative approaches have been proposed to allow learning pseudo-ensembles of models by allowing individual models within the ensemble to share parameters (Bachman et al.] 20144] Srivastava et al.) 2014: Hinton et al. 20012] Goodfellow et al.) 2013). Most notably, dropout (Hinton et al. 2012 Srivastava et al. 2014) was introduced to approx- imate the process of combining exponentially many different neural networks by "dropping out" portion of units from layers of the neural network for each batch. It was argued that this technique prevents "co-adaptation" the neural network and leads to learning more general features (Hinton)
et al. 2012).
While these techniques often improve the network generalization for i.i.d. sample sets, they are not as effective in improving the network robustness against input perturbations and in particular against adversarial attacks (Wang et al. 2018). Adversarial attacks (Goodfellow et al. 2014).. slight but carefully constructed input perturbations that can significantly impair the network's 's performance, are one of the major challenges to the reliability of modern neural networks. Despite numerous works on this topic in recent years, the problem remains largely unsolved (Kannan al., 2018 Madry et al. 2017 Zhang et al.) 2019 Sarkar et al., (202T] Pang et al.] (220): [Bashivan et al. 202T
Rebuffi et al.
2021 Gowal et al. 2021). Moreover, the most effective empirical defense methods

---

Under review as a conference paper at ICLR 2023
against adversarial attacks (e.g. adversarial training (aadry et al. 2017) and TRADES (Zhang et al., 2019)) are extremely computationally demanding (although see more recent work on reducing their computational cost (Wong et al. 2019; Shaaahi et al. (019)).
Our central premise in this work that if ensembles can be learned at the level of features (in contrast to class likelihoods), the resulting hierarchy of ensembles in the neural network could po- tentially lead to a much more robust classifier. To this end, we propose a simple method for learning ensembles of kernels in deep neural networks that significantly improves the network's robustness against adversarial attacks. In contrast to prior methods such as dropout that focus on minimizing feature co-adaptation and improving the individual features' utility in the absence of others, our method focuses on learning feature ensembles that form local "committees" similar to those used in Boosting and Random Forests. To create these committees in layers of a neural network, we introduce the Kernel Average Pool (KAP) operation that computes the average activity in nearby kernels within each layer- similar to how Average Pooling layer computes the locally averaged activity within each spatial window but instead along the kernel dimension. We show that incorpo- rating KAP into convolutional networks leads to learning kernel ensembles that are topographically organized across the tensor dimensions over which the kernels arranged. When combined with activation noise, these networks demonstrate a substantial boost in robustness against adversarial attacks. In contrast other ensemble approaches to adversarial robustness, our approach does not seek to train multiple independent neural network models and instead focuses on learning kernel ensembles within single neural network.
Our contributions are as follows:
we introduce the kernel average pool as a simple method for learning kernel ensembles in deep neural networks.
we demonstrate how kernel average pooling leads to learning smoothly transitioning kernel ensembles that in turn substantially improve model robustness against input noise.
through extensive experiments on wide range of benchmarks, we demonstrate the effec- tiveness kernel average pooling on robustness against strong adversarial attacks.
RELATED WORKS AND BACKGROUND
Adversarial attacks: despite their superhuman performance in many vision tasks such as visual object recognition, neural network predictions are highly unreliable in the presence of input pertur- bations, including natural and artificial noise. While performance robustness of predictive models to natural noise have long been studied in the literature, more modern methods have been invented in the past decade to allow discovering small model-specific noise patterns (i.e. adversarial examples) that could maximize the model's risk (Goodfellow et al. 2014).
Numerous adversarial attacks have been proposed the literature during the past decade Carlini & Wagner (2017); Croce & Hein ( Moosavi-Dezfooli et al. (2016); Andriushchenko et al. (2020); Brendel et al. (2017), 7p: Gowal et al. (2019). These attacks seek to find artificially generated samples that maximize the model'ss 's risk. Formally, given a classifier function
X' + Y. X S R",y"= = {1,..., ..,,, denote by (xx e) a perturbation function (i.e. adversarial attack) which, for a given (x, v) E X x Y, generates a perturbed sample x'' B(x, e) within the e-neighborhood of x,
B(x, <) = [x' E X': : I|x - xllp < c). by solving the following maximization problem
L(fo(t) BEB(z.c)
where L is the classification loss function (i.e. classifier's risk) and ||- is the L,p norm function. Solutions x' are called adversarial examples and are essentially the original input samples altered with additive noise of magnitude measured by the Lp norm.
Adversarial defenses: Concurrent to the research on adversarial attacks, numerous methods have
also been proposed to defend neural network models against these attacks (Kannan et al., 2018
Madry et al. 2017 Zhang et al.) 2019 Sarkar et al., [021 Pang et al. 20200 Bashivan et al. 1021
al. 2022, Rebuffi et al. 2022; 1; Gowal al. 2021). Formally. the goal of these defense methods is to guarantee that the model predictions match the true label not
Robey et al. 2021 Sehwag
only over the sample set but also within the e-neighborhood of samples x. Adversarial training.
2

---

Under review as a conference paper at ICLR 2023
which is the most established defense method to date, formulates adversarial defense minimax optimization problem through which the classifier's ' risk for adversarially perturbed samples is iter- atively minimized during training (Madry et al.] 2017). Likewise, other prominent methods such as ALP (Kannan et al. 2018) and TRADES (Zhang et al. 2019). encourage the classifier to predict matching labels for the original (x) and perturbed samples (x').
Despite the continuing progress towards robust neural networks, most adversarial defenses remain computationally demanding. requiring an order of magnitude more computational resources com- pared to normal training of these networks. This issue has highlighted the dire need for computa- tionally cheaper defense methods that are also scalable large-scale datasets such as Imagenet. In that regard, several recent papers have proposed alternative methods for discovering diverse adver- sarial examples at a much lower computational cost and have been shown to perform competitively with adversarial training using costly iterative attacks like Projected Gradient Descent (PGD) (Wong] al. 2019 Shafahi et al. 2019).
Another line of work has proposed utilizing random additive noise way to empirically improve the neural network robustness (Liu et al., 2018 Wang et al. 2018 He et al. 2019) and to derive robustness guarantees (Cohen et al.) [019:] Lecuyer et al. 2019). A Although, some of the proposed defenses in this category have later been discovered to remain vulnerable to other forms of attacks (Tamer et al. 2020), there is increasing body of work that shows the close relationship between robustness to random noise and adversarial robustness (Ford et al. 2019, Cohen et al.] 2019). Also related to proposed method, recent work on feature denoising (Xie et al.] 2019) shows that denoising feature maps in neural networks together with adversarial training leads to large gains in robustness against adversarial examples. However, this work is fundamentally different from our proposed method in that the focus of this work is on denoising individual feature maps by considering the distribution of feature values across the spatial dimensions within each feature map.
Ensemble methods have long been used in machine learning and deep learning because of their effectiveness in improving generalization and obtaining robust performance against input noise (Hastie et al.) 2009). In neural networks, eeeudo-ensemble methods like dropout (Hin- ton et al. (2012) create and simultaneously train an ensemble of "child" models spawned from a "parent" model using parameter perturbations sampled from a perturbation distribution (Bachman) (et al. 2014). Through this procedure, pseudo-ensemble methods can improve generalization and robustness against input noise. Another related method is MaxOut Goodfellow et al. (2013) which proposes an activation function that selects the maximum output amongst a series of unit outputs.
Ensemble methods:
Naturally, similar ideas consisting of neural network ensembles have been tested in recent years to improve prediction variability and robustness in neural networks with various degrees of success
(Pang et al. 2019 Kariyappa & Qureshi 2019 Abbasi al. 2020 Horváth et al. 2022 Liu al.,
2021). Ensemble adversarial training (Trmmrr et al. 2018) proposes to use adversarial examples transferred from various models during adversarial training to improve the robustness of the model. Several other works have focused on enhancing the diversity among models within the ensemble with the goal of making it more difficult for adversarial examples transfer between models (Pang eet al.) 2019; Kariyappa & Qureshi] (019)). However these ensemble models still remain prone to ensembles of adversarial attacks (Tramer et al. 2020).
3 METHODS
3.1 PRELIMINARIES
Let fo(x) X + V. where X C (1,.,, be a classifier with parameters 0. In forward deep neural networks, the classifier fo is usually composed of cascade of simpler functions (x), l E {1..... L} chained together such that the network output is computed as (x)())) For our function fo correctly classify the input patterns x,
........)
y == (() wish for it to attain a small risk for (x. y) ~ D as measured by loss function C. Additionally, for our classifier to be robust, we also wish fo to attain small risk in the vicinity of all x E X, normally defined by a Wasserstein ball around the sample points (Madry et al. 2017).
While to guarantee robustness, has to consider the maximum risk within the epsilon ball, in practice, the prediction variance can arguably be also linked to the expected robustness similar to recent work domain generalization that uses risk variance as an objective for improving model
3

---

Under review as a conference paper at ICLR 2023
generalization across domains (Krueger et al. 2021). Intuitively, model which has high pre- diction variance (or similarly high risk variance) to noisy inputs, is more likely to exhibit extreme high risks for data points sampled from the same distribution (i.e. adversarial examples). Indeed, classifiers that generate lower variance predictions are often expected to generalize better and be more robust to input noise. For example, classic ensemble methods like bugging, boosting, and dom forests operate by combining the decisions of many weak (i.e. high variance) classifiers into stronger one with reduced prediction variance and improved generalization performance (Hastie et al. 2009).
Given an ensemble of predictor functions fi, I..... K with zero or small biases, the ensemble prediction (normally considered as the mean prediction yj * Σx' 31) reduces the expected gen- eralization loss by shrinking the prediction variance. To demonstrate the point, one can consider K i..d. random variables with variance o² and their average value that has variance of o² Based on this logic, one would expect ensembles of neural network classifiers to be more robust in the pres- ence of noise or input perturbations in general. However, such ensemble models have been shown to remain prone to ensemble of adversarial attacks with large epsilons (Tramer et al.. 2020).
We reasoned that individual networks participating these ensembles may still learn different sets of non-robust representations leaving room for the attackers to find common weak spots across all individual models within the ensemble. At the same time, constructing ever-larger ensemble classifiers might quickly become infeasible, especially in the of neural network classifiers. On the other hand, learning robust features has been suggested as a way towards robust classification (Bshivva al. 2021). Consequently, if individual kernels within single network robust, it would become much more difficult to find adversaries that can fool the full network. the next section, introduce the Kernel Average Pool way towards learning ensemble kernels with better robustness properties against input perturbations.
3.2 KERNEL AVERAGE POOL (KAP)
Mean filters (a.k.a,, average pool) are widely accepted as simple noise suppression mechanisms in computer vision. Spatial average pooling layers are commonly used in modern deep neural networks (Zoph et al. 2018) by applying mean filter along the spatial dimensions of the input to reduce the effect of spatially distributed noise (c.g. adjacent pixels in an image).
Here, we wish to substitute each kernel in the neural network model with an ensemble of ker- nels performing the same function such that the ensemble output is the average of individual ker- nel outputs. This can be conveniently carried out by applying the average pool operation along the kernel dimension of the input tensor.
O
In
OeN
h
Given an input = E DxNx, where and N₂ de- note the input dimension and the number of ker- nels respectively, the kernel average pool opera- tion with kernel size K and stride S, computes the function
Figure 1: Schematic of one-layer neural network with (right) and without (left) the kernel average pooling operation.
Sk
K
Za
K
1= Sk
Importantly, when the output of an operation linear with respect the weights on an input (e.g. linear layers or convolutional layers), KAP is functionally equal to computing the locally averaged weights within the layer and could be interpreted as form of Kernel Smoothing (Wang et al. 2020) conditioned that the nearby kernels are more or less similar to each other.
Sk+
Sk+
Z
W,F
W
1=
Moreover, the degree of overlap (i.e. parameter sharing) across kernel ensembles can be flcxibly controlled by adjusting the KAP stride. Choosing stride S = K, produces independent kernel en-
4

---

Under review as a conference paper at ICLR 2023
sembles (no parameter sharing between ensembles), while having stride S < K enforces parameter sharing across kernel ensembles.
Eq. 2 assumes that kernels are arranged along one tensor dimension. However, KAP could generally be applied any D- dimensional tensor arrangement of kernels. For example apply 2-dimensional KAP (mainly considered in our experiments) on the input 2, one can first reshape the channel dimension into a 2- 2-dimensional array and then apply KAP along the two kernel dimensions (see A and Alg. 1). Importantly, using higher dimensional tensor arrangements of kernels when applying KAP, allows parameter sharing across a larger number kernel ensembles.
Algorithm 2D Kernel Average Pool
Input: layer input x, kernel size k, stride 8, vectorization operator Vec and its inverse Vec zero padding function Pad, and average pooling function AvePool.
h < Vec VD. DD,VD.WWH (Vec(x*))
h < Pad(h, k=1)
AvePool k.
t Veew, H, (Ve((h"))
return:
In the next two subsections, we will explain how training networks with KAP-layers leads to learning topographically organized kernel ensembles ($.. and how these kernel ensembles may contribute to model robustness $3.4
3.3 KERNEL AVERAGE POOLING YIELDS TOPOGRAPHICALLY ORGANIZED KERNEL ENSEMBLES
Consider a simple neural network with one hidden layer and N₂, units (eft( where the hidden unit activation is h; Wi50 and the network output is computed as
y
Waah;
(4)
In this network, the output gradients with respect weight parameters be computed as
Oy 0wn
Oy
W222
W1:
(5)
awz
Now consider variation of this network where the hidden unit activations are passed through kernel average pool with kernel size K (Fig rightt) In the KAP-network, the output gradients with respect weight parameters are altered such that
Oy Owi
Oy Owz
Σ I= R-1
W2
WHF
K
K
I= R-1
where, to simplify the limits, K is assumed to be an odd number. In contrast to the regular network, in the KAP-network, the gradients of the output with respect to the incoming (wi)) and outgoing (W2s) weights in node depend on the average of outgoing and incominng weights, respectively, over the kernel average pool window. The difference in the output gradients with respect to weights wis (and similarly for W2;) for nodes and = i + d can be written as
Oy
Oy Owij
Σ
Σ
W2
W2p
Ow
1=i-d- K-1
From 7 clear that when K large K > 1. the difference in the output gradients with respect weights for pair of nodes (i.j) depends the absolute difference between node indices (li - jD and smaller for a pair of nodes with smaller index difference (i.e. physically closer nodes). Thus, when training with backpropagation, weights connected physically closer nodes (in
5

---

Under review as a conference paper at ICLR 2023
terms of their index numbers) will likely receive more similar gradients compared to those that farther. Consequently, these physically closer weights are more likely to converge to more similar values. On the other hand, for a KAP with stride=S and kernel size K, each kernel participates in 13 ensembles. Kernel sharing between ensembles provides a natural mechanism preventing the participating kernels in each group from converging to the exact same parameter values. Our empirical results show that the interaction between these two forces leads to smoothly transitioning topographically organized kernel maps (Fig 3).
3.4 KERNEL AVERAGE POOLS AND ADVERSARIAL ROBUSTNESS
Our approach is closely related to randomized smoothing (Lecuyer et al., 2019 Cohen et al. 2019). which for a sample (xr, y) consists of learning to predict y with higher probability than other classes over N(x, o² I). When combined with noise resampling to estimate the true class, randomized smoothing yields certifiable robustness against adversarial perturbations. More recently, Horváth et al. (2022) show that in addition injecting noise in the input, averaging the logits over an ensem- ble of models leads to a reduction of the variance due to the noisy inputs in randomized smoothing and in turn improves the certified robustness radius. They propose to learn ensemble
(zz) = Ens(fe(x + m)) = 1. Σ
Σ
(8)
where n o2) and f. denotes the l-th presoftmax classifier in ensemble of L classifiers.
Similar to randomized smoothing [eccyyer et al. 2019 Cohen et al. 2019), by introducing stochas- ticity in the input during training as in randomized smoothing, we hope to learn model that is robust to perturbations. Moreover, in a simple setting where only one layer of KAP is used without activations, we note that KAP averages multiple filters before applying them the input of KAP block (eq. 33. This can be understood as averaging the features obtained from multiple models, each Unlike|Hovvthh corresponding to filter. Unlike Horváth al. (2022), our ensembling done the level of features
instead of the logits. Nevertheless, their arguments still apply to the case of reduction of variance in the features computed. Therefore, if KAP features are used as inputs to fully connected layers to compute logits, reduction of variance in the features will translate into reduction of variance in the logits. An additional key difference that in our case, in the full KAP-based models used in $4 we use networks consisting of multiple cascaded KAP blocks extract features, similar to how multiple convolutional blocks are used sequentially convolutional networks. Each KAP block consists of an ensembling, from input that was perturbed with Gaussian noise. In other words, our approach can be understood as recursively performing a form of Horváth et al. (002)) 's ran- domized smoothing with ensembles, by interpreting the output of each KAP block randomized smoothing input for the next block. In this light, our KAP architectures perform the operation
fe o 9N. o o 91 91(z) where ga (I) = Ens[f + n2)]
(9)
where the ensembling is performed by the KAP operation, potentially including activation functions, Ni is the number of KAP blocks, n, is Gaussian noise sampled in KAP block i, fi is the operation (e.g. a convolution) performed in KAP block i before a kernel average pool, and fe maps the representations to the logits of the classes. Our reasoning for this recursive approach is twofold: first, during training, this encourages all layers to learn to be robust to variance in their input. Otherwise in deep networks, some layers may not be exposed to significant variance in their input depending on their depth, due to the variance reduction occurring in the earlier layers. Second, we hope that by having randomized smoothing at every KAP block, the model will require less perturbed inputs. If this approach successfully performs randomized smoothing on the features, we may hope that this will lead the representations of adversarially perturbed inputs to remain close to the distribution of inputs perturbed with Gaussian noise, on which the network has been trained and therefore should perform well. We empirically verify this intuition in the appendix (Fig A5)
4 EXPERIMENTS
In this section we empirically demonstrate the effectiveness of our proposed kernel average pool operation in boosting robustness in deep neural networks.

---

Under review as a conference paper at ICLR 2023
4.1 EXPERIMENTAL SETUP
Datasets: We validated our proposed method on several standard benchmarks CIFAR I0, CIFAR100
(Krizhevsky 2009). Tinylmagenet Le & Yang (2015), and Imagenet (Deng al., 2009). We used standard preprocessing of images consisting of random crop and horizontal flipping for all datasets. We used images of size 32 in our experiments on CIFAR datasets, 128 for Imayemagene,, and 224 for Imagenet.
Baseline models: We compared the results from our proposed model to several baselines including (i) the original ResNet18 (i.e. without additional KAPs) trained normally (marked with NT); (ii) the original ResNet18 with additive activation noise (marked with NT(o = )); (iii) the original ResNet18 trained using adversarial training with early stopping (marked with AT-ES).
Adversarial attacks: We assessed the model robustness against various adversarial attacks in- cluding Lox Projected Gradient Descent (PGD) (Madry et al.) 2017). SQUARE black-box (An- driushchenko et al. 2020) attack and the AutoAttack (Croce Hem 2020). Notably, AutoAttack is ensemble attack consisting of four independent attacks and is considered one of the strongest adversarial attacks in the field. We used с value of 0.031 for experiments on CIFAR datasets and 0.016 for TinyImagence and Imagenet datasets. See supplementary Table A3 for full details of tacks used for model evaluation. Importantly, we applied each attack on the model without activation noise to prevent activation noise from potentially masking the gradients the network.
Training and evaluation considerations: In our experiments, we primarily used the ResNet18 architecture (He al., 2016) that consists four groups of layers and each group containing two basic residual blocks. For the KAP variations of ResNet18, 8, we added the KAP operation after each convolution in the network. In variations of the model where we introduced activation noise, we added random noise sampled from the Gaussian distribution (0, o²) after each KAP operation (and after each convolution the original model architecture).
For adversarial training of the baseline AT models on CIFAR1O, CIFARI00, and TinyImagenet datasets, we used the normal adversarial training procedure with early stopping and T.go PGD attack (Madry al. 2017 Rice et al. 2020). We used 20 iterations for CIFAR training runs and 10 iterations for Tiny Imagenet. On Imagenet dataset, we used Fast-AT method (Wong et al. 2019) with the default training parameters consisting of three training phases with increasing image resolution.
4.2 ROBUSTNESS AGAINST STANDARD ADVERSARIAL ATTACKS ON CIFAR
Table 1: Comparison of adversarial accuracy against various attacks on CIFAR1O and CIFAR100 00 datasets. For all attacks used 0.031. All attacks are performed the corresponding model
without input or activation noise.
DATASET
MODEL
CLEAN
PGD-Loc
AUTOATTACK
SQUARE
RNI8-NT
94.66
0.0
0.0
0.87
RNI8-NT (a
0.1)
88.95
9.69
8.90
61.7
WRN- 4(-BE)) WEN ET AL.
2020
95.60
7.90
7.80
21.00
CIFAR1O
RNI8-AT-ES ET AL 2020
84.20
43,70
43.00
49.10
KAI8-KAP-NTE
U.1.K K
79.09
67.7
41.8
44.49
RNI8-KAP-NT (o 0.2, K
74.30
65.1
44.40
47.5
RN RNI8-NT (a = 0.1)
74.00
0.0
0.0
0.20
61.60
6.00
5.20
33.70
CIFAR100
RNI8-AT-ES (RICE ET AL.
2020
56.50
20.40
19.60
22.86
RNIS-KAP-NO KAP-NTTO U.I.A.
48.20
33.9
15.70
17.40
RNI8-KAP-NTC 0.2. K 3)
38.60
31.9
17.60
27.40
We first compared robustness in convolutional neural networks with and without KAP on the CI- FAR10 and CIFARIOR datasets. For this we trained the vanilla ResNet18 architecture and two variations of this architecture where all convolution operations were followed by KAP. To make sure that the stochasticity due to activation noise does not interfere with gradient estimation during attacks, we performed all attacks on the corresponding model without input or activation noise.
Table lists the robustness of different models trained CIFAR10 and CIFAR 100 datasets against several commonly used adversarial attacks. Confirming prior work (He et al. 2019), we found that on both CIFAR10 and CIFAR 100 datasets, training the network with noisy activations improve

---

Under review as a conference paper at ICLR 2023
the network robustness. This improvement was most noticeable against the strong SQUARE black- box attack and to a lesser degree against PGD and AutoAttack. Furthermore, we found that adding KAP to the model significantly improves the robustness against all attacks and even beyond that of the adversarially trained ResNet18 architecture (AT). Robustness remained high against adversarial examples that were generated using models with activation noise ($A.2) as well as those transferred from RNI8-NT and RN18-AT models (TabldA5). The improved robustness was however at the expense of noticeable reduction in clean accuracy that also increased with the activation noise variance a. Although, it should also be noted that KAP models were trained using normal training procedures (i.e., no adversarial training) and as a result the computational cost of training was only fraction of that required for adversarial training (~ 0. 0.14%). See the supplementary Table A4 for comparison of training speed between KAP models and adversarial training.
We additionally explored the effect of kernel pooling type (No pooling, Max pooling, and Average pooling) and activation noise variance (ar E 0.1, 0.2]) on the robust accuracy with the ResNet18 architecture. To get a better sense of how robustness generalizes to higher-strength attacks (i.c. larger e), we also tested each model on several e values ( to 255
32 using AutoAttack (Fig. 2). We
255
found that (a) adding KAP to RN18 without activation noise already makes the network substan- tially more robust against attacks with smaller epsilons (Fig. 2b); (b) KAP model showed strong robustness to adversarial attacks on par with RNI8-AT and even better performance against attacks with larger (Fig. Db), while the variation with Kernel Max Pool was the least robust variation; c) larger activation noise variance during training led to higher robustness against stronger attacks (Fig. 2p. Furthermore, in separate experiments, we also investigated the effect of model depth and kernel ensemble size on robustness, which we report in the appendix $A.3] We found that increas- ing the network depth and KAP kernel size both substantially improve the network robustness to AutoAttack.
4.3 ROBUSTNESS AGAINST ADVERSARIAL ATTACKS ON IMAGENET
To test whether our results scale to larger datasets, we also trained and compared convolutional neural networks on two large-scale datasets, namely Tinylmagenet (200 classes) and Imagenet (1000 classes). Here again, we used the & architecture as our baseline and created variations of this architecture by adding KAP after every convolution operation. We found that reducing the weight decay parameter when training the KAP networks on Imagenet improves the performance of the KAP models and used a weight decay of 1e-5 in training our best models on Imagenet. In addition to the PGD-Loe Loe attack, we also evaluated the models using AutoAttack on these datasets. However, because of its high computational cost, we used 1000 random samples from the validation set.
Table 2: Comparison of robust accuracy against various attacks on Tinylmagenet and Imagenet datasets. For all attacks we used e 0.016. All attacks are performed on the corresponding model without input or activation noise. i: models trained using Fast Adversarial Training (Wong et al. 2019)
DATASET
MODEL
CLEAN
PGD-
AUTOATTACK
SQUARE
RNI8-NT
58.90
0.00
0.00
3.30
RNI8-NT (o 0.1)
56.50
1.90
1.80
35.50
TINYIMAGENET
RNIS-AT-ES (RICE
2020
45.80
25.40
21.60
29.10
RNI8-KAP-NTTE U.I.A
39.60
25.70
16.5
18.70
RNI8-NT RNIS-NT (o 0.1)
68.68
0.0
0.0
2.80
$1.20
0.19
0.20
6.70
RN18-AT!
53.20
8.00
8.20
IMAGENET
RNI8-KAP-NT (a 0.1,K
9.60
6.05
2.85
2.95
RNI8WIDEX4-AT
62.00
27.81
11.80
14.10
RNISWIDEXA KAP-NN (a = K
38.00
31.49
15.3
14.40
Table 2 summarizes the robust accuracy of different models on these two datasets. Overall, we found that (a) on these two datasets, sole usage of the activation noise similar to (He et 2019) was much less effective at improving the network robustness; (b) robustness in the KAP variation of ResNetl8 18 was significantly better than the orignal network and baseline trained with noise but slightly lower than the adversarial trained network on Imngnneetet dataset. On Imagenet, we found that the KAP variation of ResNet18 model was struggling learn the task completely, reaching only about 10- 12% accuracy on the clean dataset. We reasoned that the difficulty in learning the task on this dataset might be due to the large number of classes in this dataset and the possibility that the ResNet18
8

---

Under review as a conference paper at ICLR 2023
model does not have sufficient capacity to learn enough kernel ensembles to tackle this dataset. For this reason we also trained a wider version of RN18 in which we multiplied the number of kemnels in each layer by factor of 4 (dubbed RN18WideX4). We found that this wider network significantly boosted the robust accuracy on Imagenet, even surpassing the adversarially trained model. However, again we found that this boost adversarial robustness is accompanied by significant decrease in clean accuracy.
Architecture
Noise std
Architecture
accuracy
20-
RN18-KAP
accuracy
50-
RN18-KMP
Bccuracy
50-
0.0
0.1
RN18 2PP8
0.2
RN18
AT
0-
0-
0-
0.05
0.10
0.05
0.10
0.05
0.10
epsilon
epsilon
epsilon
(a)
(b)
(c)
Figure 2: Robust accuracy in RN18-KAP model against AutoAttack with various attack strength e on CIFARIO dataset. (a) RN18 and AN18-KAP (a 0, K 3): (b) RN18, RN18-KAP, RNI8-
KMP (a = 0.1, K = 3) and AT; (c) RNI8-KAP (K 3) for various noise levels σ.
4.4 KAP YIELDS TOPOGRAPHICALIE ORGANIZED KERNELS
As described in the methods, KAP combines the output of multiple kernels into single activation passed from one layer to the next. As this operation creates dependencies between multiple kernels within each group, we expected certain level of similarity between the kernels within each pseudo- ensemble to emerge. To examine this, we visualized the weights in the first convolutional layer of the normally trained (NT), the adversarially trained RNI8 architecture (AT) and two variations of RNI8 with Kernel Average Pool (KAP) and Kernel Max Pool (KMP) on CIFARIO and Imagenet datasets (Fig. 3) As a reminder for these experiments, we had incorporated a 2-dimensional KAP that was applied on the kernels arranged on a 2-dimensional sheet.
As expected, the kernels in NT and AT models did not show any topographical organization. Kernels in KMP model were sparsely distributed, with many kernels containing very small weights. This was presumably because of the competition amongst different kernels within each pseudo-ensemble that had driven the network to ignore many of its kernels. In contrast, in KAP model, we observed an overall topographical organization in the arrangement of the learned kernels along the two di- mensional sheet. Moreover, in many cases the kernels gradually shifted from the dominant pattern in one cluster to another traversing along either of the two kernel dimensions. This topographical organization of the kernels on the 2-dimensional sheet is reminiscent of the topographical organi- zation of the orientation selectivity of neurons in the primary visual cortex of primates (Hubel & Wiesel 1977).
C10-NT
C10-KMP
C10-AT
C10-KAP
Imagenet-AT
Imagenet-Ka KAP
Figure 3: Visualization of the learned weights in the first layer of several variations of RN18 model.
CONCLUSION
We proposed Kernel Average Pooling mechanism for learning ensemble of kernels in layers of deep neural networks. We showed that when combined with activation noise, KeP-networks form process that can be thought of as recursive randomized smoothing with ensembles applied at the level of features, where each stage consists of applying ensemble of kernels followed by noise injection. Our empirical results demonstrated significant improvement in network robustness

---

Under review as a conference paper at ICLR 2023
at fraction of computational cost of state-of-the-an methods like adversarial training. However, because of the need for learning ensemble of kernels at each network layer, the improved robustness is often accompanied by reduced performance the clean datasets. Our results suggest feature- level ensembling as a practical and scalable approach for training robust neural networks.
REFERENCES
Mahdieh Abbasi, Arezoo Rajabi, Christian Gagné, and Rakesh B Bobba. Toward adversarial robust- ness by diversity in an ensemble specialized deep neural networks. Canadian Conference on Artificial Intelligence, pp. 1-14. Springer, 2020.
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey Hinton. Neural Additive Models: Interpretable Machine Learning with Neural Nets. In NeurlPS, 2021.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. In European Conference Computer Vision, pp. 484-501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of se- curity: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274-283, PMLR, 2018.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. Advances in neural information processing systems, 27:3365-3377 2014.
Pouya Bashivan, Reza Bayat, Adam Ibrahim, Kartik Ahuja, Mojtaba Faramarzi, Touraj Laleh, Blake Richards, and Irina Rish. Adversarial feature desensitization, Advances Neural Information Processing Systems, 05110665-10677 2021.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-basee adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint 17X.027712 2017.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. Pro- ceedings IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
Jeremy Cohen, Flan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. 36th International Conference on Machine Learning, ICMI. 2019, 2019-Jne 2356, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Nicolas Ford, Justin Gilmer, Nicholas Carlini, and Ekin D. Cubuk. Adversarial examples are natural consequence of test error in noise. volume 2019-June, pp. 4115 2019.
Yoav Freund. Boosting weak learning algorithm by majority. Information and computation, 121 (2):256-285, 1995.
Yoav Freund, Robert E Schapire, al. Experiments with boosting algorithm. Citeseer, 1996.
Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International conference on machine learning, pp. 1319-1327. PMLR, 2013.
10

---

Under review as a conference paper at ICLR 2023
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang, Timothy Mann, and Pushmeet Kohli. An Alternative Surrogate Loss for PGD-based Adversarial Testing. 2019.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg. Dan Andrei Calian, and Timothy Mann. Improving robustness using generated data. 2021. URL http://arxiv.
org/abs 221002110,.0 09468
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learnin. Cited on, pp. 33, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Zhezhi He. Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 588-597, 2019.
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdi- nov. Improving neural networks by preventing oo-adattttion of feature detectors. 2012.
Miklós Z Horváth, Mark Niklas Müller, Marc Fischer, and Martin Vechev. BOOSTING RANDOM- IZED SMOOTHING WITH VARIANCE REDUCED CLASSIFIERS. pp. 33, 2022.
David Hunter Hubel and Torsten Nils Wiesel. Ferrier lecture-functional architecture of macaque monkey visual cortex. Proceedings of the Royal Society of London. Series B, Biological Sciences, 198(1130):159, 1977.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:/803.06337, 2018.
Sanjay Kariyappa and Moinuddin K Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901.00 009981 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
David Krueger, Ethan Caballero, Joern Hennik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. -uttof-disttibuttion generalization via risk extrapo- lation (rex). In International Conference Machine Learning, pp. 5815- PMLR, 2021.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7,7)33,000 2015.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 656-672. IEEE, 2019.
Chizhou Liu, Yunzhen Feng, Ranran Wang, and Bin Dong. Enhancing certified robustness via smoothed weighted ensembling, 2021. URL http:://aa..w...gg/a/b////////// /005/ 09363 type:
article.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho Jui Hsieh. Towards robust neural networks via random self-ensemble. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11211 LNCS:381-397, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: /70.000836...33 2017.
Seyed Mohsen Moosavi- -ezzooll,,,,,,,,,,,.o Alhussein Fawzi, and Pascal Frossard. DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2016-Deccm:2274-22232 2016.

---

Under review as a conference paper at ICLR 2023
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. 36th International Conference on Machine Learning, ICML 2019, 2019-Junc:8759-8777 2019.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Kun Xu, Jun Zhu, and Hang Su. Boosting adversarial training with hypersphere embedding. Advances in Neural Information Processing Systems. 2020- December(NeurIPS), 2020.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation can improve robustness. 2021. URL htth:t//aaaxiy>> org/ /bs/
2111 05328
Leslic Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. Par:F16614:8049 -0744 2020. ISBN: 9781713821120.
Alexander Robey, Luiz F. O. Chamon, George J. Pappas, Hamed Hassani, and Alejandro Ribeiro.
Adversarial Robustness with Semi-Infinite Constrained Learning. Neur/PS, pp. 1-18, 2021.
Anindya Sarkar, Anirban Sarkar, Sowrya Gali, and Vineeth Balasubramanian. Get Fooled for Right Reason: Improving Adversarial Robustness through Teacher-guide Curriculum Learning Approach. NeurlPS, 2021.
Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? pp. 1-30, 2022.
Ali Shafahi, Mahyar Najib,, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems, 32(Neur)PS), 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learn- ing Research, 15: 1958, 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, lan Goodfellow, Dan Boneh, and Patrick Mc-
Daniel. Ensemble adversarial training: Attacks and defenses. 6th International Conference Learning Representations, ICLR 2018 Conference Track Proceedings, 2018.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002. 08347, 2020.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High- figh-firquency component helps plain the generalization of convolutional neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8681-869) IEEE, 2020. ISBN 978-1- 72817-168-5. doi: 10. 00.10009CCP42260000000000011 URL / Leee. org/ htc//accument99154255
Siyue Wang. Xiao Wang, Pu Zhao, Wujie Wen, David Kaeli, Peter Chin, and Xue Lin. Defensive dropout for hardening deep neural networks under adversarial attacks. In Proceedings of the International Conference on Computer-Aided Design, pp. 1-8, 2018. doi: 10. 10.1145/3240765. 3264699. URL httptt/////axiii org hwt.gg/ab////89.. 00..055666
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: An alternative approach to efficient ensemble and lifelong learning, 2020. (Rtp:ttt::/arxxiv.oo http:///as/0002002..
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training.
In International Conference on Learning Representations, 2019.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accu- racy without increasing inference time. 2022.
12

---

Under review as a conference paper at ICLR 2023
Cihang Xie, Yuxin Wu, Laurens Van Der Maaten, Alan L. Yuille, and Kaiming He. Feature denois- ing for improving adversarial robustness. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 201--Jun:5501-500, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning. pp. 7472-7482. PMLR, 2019.
Zhi Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: Many could be better than all. Artificial Intelligence, 137:239-263, 2001. doi: 10.101//j.rrtin.222 10.00...01.....
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.I 2-DIMENSIONSAL KAP
Given an input 2 E DxN, where = N, x Nr is the number of kernels that can be rearranged into N, rows and No columns, and D denotes the input dimension, the 2D kernel average pool operation with kernel size K K and stride S, computes the function
=
(Sk N.)+ Kig
2sk (x)
Σ
Σ
2(([N+m))
(10)
K
(Sk No) K
Graphically, this procedure is visualized in Fig.
Kernel Average Pool
Average Kernel
everage
pool
Input
Output
reshspe
reshape
Kerns Dimensioo Spatial Dimension
(a)
(b)
Figure A1: Graphic illustration of 2-dimensional KAP. a) the input tensor is first reshaped such that the spatial dimensions are collapsed onto single dimension and the kernel dimension is rearranged as matrix. A 2D average pool is applied on the reshaped tensor, and the resulting tensor is reshaped back into its original shape; b) the average kernel is applied per spatial position (i.e. pixel) and computes the average of nearby kernel values at that spatial position.
A.2 CLASSIFICATION ROBUSTNEESS FOR DIRECT ATTACKS ON FULL MODELS WITH ACTIVATION NOISE
For completeness of our assessments, we also considered the case where the attacker is applied to the full model with activation noise. Tables AT and A2 summarize the robust accuracies on CIFAR 10, CIFAR100, Tinylmagenet, and Imagenet datasets. Compared to the case where the attackers were applied on the model without activation noise, the KAP models achieved higher robust accuracies against attacks performed on the full model.
We reasoned that this improvement in robustness is potentially because of the test-time stochas- ticity in layer activations and its possible detrimental effect gradient estimation in attacks that only use a single sample. To further investigate this, we also tested all of our models against the random version of AutoAttack (AutoAttack,... in addition the standard version of AutoAt- tack Attakkttt This attack applies Expectation over Transformation introduced by (Athalye) al. 2018) correctly compute the gradients over the expected transformation the input (apgd- ce and apgd-dlr attacks, each with 20 iterations). We found that KAP models were also equally robust against the random version of AutoAttack.
A.3 EFFECT OF NETWORK DEPTH AND KERNEL ENSEMBLE SIZE ON NETWORK ROBUSTNESS
We performed three additional experiments to investigate the effect of network depth, number of KAP layers, and kernel ensemble size (which controlled by KAP kernel size) on the network robustness. In the first experiment, varied the network depth (number of convolutional blocks)
14

---

Under review as a conference paper at ICLR 2023
Table A1: Comparison of adversarial accuracy against various attacks CIFARIO and CIFAR100 datasets where each attack is performed on the full model with activation noise. For all attacks used e 0.031. AutoAttackma results for stochastic models reported for 20 and 50 (in parentheses) EoTs
PGD-L.
AUTOATTACK,o
AUTOATTACK.od
SQUARE
NN RNIS-NT 0.1)
94,66
0.0
0.0
0.1
0.87
10.91
(6,20)
84.42
CIFARIO
RNI&-AT-ES RNIS-KAP-TT ta -01.K =3)
84.20
43.70
43.00
43.38
49.10
79.09
67.55
58.40
61.64 (56.1)
66.12
KNI&-KAP-NT 0.2.K (3))
74.30
61.82
55.19
57.04 (57.80)
63.64
RNI8-NT
74.00
0.0
0.0
0.0
0.24
RNI8-NT (a RNIS-AT-ES
61.60
5.23
4.06
4.30 (4.40)
49.46
CIFARIO0
56.50
20.40
19.60
19.83 88
22.86
RNI8-KAP-NT 0 (3))
48.20
36.08
23.70
30.52 (28.20)
RNI8 KAP-NT (a 02, (3))
38.60
31.49
23.36
31.20 (31.00)
27.40
Table A2: Comparison of robust accuracy against various attacks on TinyImagenet and Imagenet datasets where each attack is performed on the full model with activation noise. For all attacks we used 0.016. t: models trained using Fast Adversarial Training (Wong et al. 2019).
DATASET
MODEL RNI8-NT RNIS-NT (ar - 0.1)
CLEAN
PGD-Loo
AUTOATTACK,
AUTOATTACK
0.12
0.02
0.04
56.50
0.26
0.16
0.38
TINYIMAGENET
RNI8
RNI8-KAP-NT @ O.1,K (3))
39.60
27.24
20.78
22.50
RNIB-NT RNIS-NT (ar 0.1)
0.0
51.20
0.19
0.20
0.0
IMAGENET
RNI&-KAP-NT 0,1 (3))
RNIS-ATI
53.20
9.60
19.23
9.14
6.00
3,00
4,80
6.00
RNI8WIDEX4-AT!
62.00
27.81
11.80
11.60
RNISWIDEXA KAP (a O.I.K (3))
38.00
30.91
21.40
31.00
between 3 layers while keeping the network width fixed. We observed that increasing the network depth improves its robustness against AutoAttack for both KAP and non-KA KAP networks. However, this improvement
much more pronounced for KAP networks (Fig. A24)
Since the number of convolutional and KAP layers were co-varying in the previous experiment, solely due to the increasing
possible that the observed improvement in network robustness network depth. To investigate this, in the second experiment, we kept the network depth fixed while changing the number KAP layers. We trained the following three variations of 3-layer CNNs: 1) one KAP layer after first convolutional layer; 2) one KAP layer after first and second convolutional layers; 3) KAP layer after each of the three convolutional layers. In all architectures, the KAP layer consisted of 3 x 3 kernel with stride 1. Activation noise with o 0.1 was added after each KAP layer. We validated the adversarial accuracy of each of these models with AutoAttack- with varying epsilon (Fig. A2 b). We observed that while the number of convolutional layers was fixed, adding more KAP layers consistently improved the network robustness against AutoAttack.
In the third experiment, we varied the KAP kernel size between to 4 to investigate the effect of larger ensemble sizes on the network robustness. each network, we set the KAP stride equal to the KAP kernel size to avoid overlapping between ensembles, and also increased the network width by the same factor as the KAP kernel size to allow each network to learn the same number of kernel ensembles. We found that increasing the KAP kernel size and consequently the ensemble size further improves the network robustness against AutoAttack (Fig, A2b). The resulting kernel ensembles in each of these models shown in Fig A3
15

---

Under review as a conference paper at ICLR 2023
acuracy
40-
30-
Layer
KAP
w/O
80c (ocuresy
30
BayerCNN 2KAP
SayerCMN-1KAP
BayorCNN-3KAP
accuracy
60-
Wisth Factor
20-
Architecture
40-
layer CNN
10-
3 layar CNN
layer
10
20-
0.02
0.04 epsilon
0.06
0.02
0.03
0.04
0.05
0.00
0.05
epsilon
epsilon
(a)
(b)
(c)
Figure A2: Effect of network depth and kernel ensemble size on the robustness to AutoAttack
(e= 0.031) on CIFARIO dataset. a) increasing the number layers in CNN from 1-3 significantly improves its robustness to AutoAttack in networks with KAP compared to networks without KAP;
b) increasing the number of KAP layers while keeping the number convolutional layers fixed (3) improves robustness to AutoAttacc;e) increasing the kernel ensemble size by increasing the KAP kernel size (expansion) improves the network robust accuracy against AutoAttack. All attacks are performed on the corresponding model without noise.
K=1
K=2
K=3
K=4
Figure A3: Visualization of the learned weights the first layer of 3-layer convolutional networks with varying KAP kernel size K.
Table A5: Robust accuracy against transfer attacks on CIFARIO. Attacks were generated using each Reference Model and tested on the Model. For all attacks we used 0.031.
MODEL
REFERENCE MODEL
AUTOATTACK
RNI8-NT
76.4
RN18-KAP-NT (a 0.1, K K= 3)
RN18-AT
66.19
RN18-NT
68.6
RNI8-KAP-NT (o 0.2, =3
RNI8-AT
61.84
16

---

Under review as a conference paper at ICLR 2023
Block1
Block2
Block3
Block4
100
50
0.0
0.5
0.0
0.5
activation
activation
activation
activation
2
400
noise
1.0
adversarial
1
5
200
0.5
0.0
0
0
0
0 2
0
0.0
0.5
activation
activation
activation
activation
Figure A4: Distribution of layer activations in a random kernel for RNI8-NT(O 0.1) (top) and RNI8-NT-KAP(G = 0.1) (bottom) trained on CIFAR10 dataset. We extracted the layer activations response to a single random input image perturbed 100 times with random Gaussian noise or AutoAttack. Distributions of layer activations to noise and adversarial examples exhibit increased divergence in Block 4 in RNI8-NT model but not in RN18-NT-KAL The same pattern is replicated when considering other randomly selected input images from the validation set.
Block1
Block2
Block3
Block4
Logits
0.4
0.4
1.0
0.2
0.2
0.2
0.5
0.2
0.0
0.0
0.0
0.0
0.0
100
150
60
70
27. 27.50..0
50
75
30 40
distance
distance
distance
distance
distance
0.04
noise
adversarial
0.05
0.1
0.2
0.2
0.02
0.00
8.00
0.0
0.0
0.0
300
400
150175 distance
901 10
35 40 45
12.55.07.5 distance
distance
distance
distance
Figure A5: Distribution of perturbation magnitude in layer activations for RN18-NT(O = 0.1) (top) and RNI8-NT-KAP(C 0.1) (bottom) trained CIFAR10 10 dataset. We extracted the layer activa- tions in response to single random input image perturbed 100 times with random Gaussian noise Attokttack. Perturbation magnitude is computed the Tog distance between perturbed and clean input activations (f n:) f() for noise and (xr() (()( for adversarial perturba- tions). Distributions layer activations to noise and adversarial examples exhibits divergence in
later blocks in RNI8-NT model but not in RNIS-NT-KAP.
17

---

Under review as a conference paper at ICLR 2023
Table A3: Attack hyperparameters used to validate model robustness on each dataset.
Attack
Dataset
Steps
Size (c)
More
CIFAR
2/6
step 24o
PGD-Toc
TinyImagenet
20
step= 253
24s
Imagenet
step= 255
755
CIFAR
default standard AA APGD-ce, APGD-t, FAB,
24
Aulokklack.ad std
TinyImagenet
100
245
Imagenet
SQUARE
255
CIFAR
default random AA APGI-ce and APGD-dIr, each with 20 EoT iterations
255
AutoAttackend
Tinylmagenet
100
21
Imagenet
255
CIFAR
2
SQUARE
TinyImagenet
5000
default SQUARE setting
24
Imagenet
255
Table A4: Comparison training speed between alternative models. All training times were com- puted on the CIFAR 10 dataset and ResNet18 88 architecture using single A100 GPU.
Dataset
Model
Ave. Training Speed / Epoch
RNI8-NT
12.5 + 0.5 sec
CIFARIO
RNI8-AT
100.75 sec 14.5 + 0.5 sec
RNI8-KAP-NT (a 0.1, 3)
RN18
1.0
RN18-KAP
Ayn
AX
0.5
0.0
layer1
layer2
layer3
layer4
linear
Figure A6: Normalized magnitude of change in each layer's activations in response to adversarial perturbations (Iladd Wetall for AutoAttack L2, 1. CIFAR 10 dataset for RN18-NT (a 0.11)
|[aall and RNI8-KAP-NT 0.1).
18

---

Under review as a conference paper at ICLR 2023
Architecture
Noise
accuracy
Architecture RN18
accuracy
RN18
accuracy
5-
RN18-KAP
0.2
RN18-KAP
AT
AT
0-
3
epsilon
epsilon
epsilon
(a)
(b)
(c)
Figure A7: Robust accuracy in RN18-KAP model against AutoAttack-LL2 with various attack
strength on CIFARIO dataset. (a) RNI8 and RNI8-KAP (a 0, K 3); (b) RNI8, RNI8-
KAP, RNI8-KMP (a = 0.1, K 3) and AT; (c) RNI8-KAP (K = 3) for various noise levels o.
Certifeid Accuracy
RN18
0.75
RN18-KAP
0.50
0.25
0.1
0.2
0.3
0.4
Noise o
Figure A8: Certified accuracy of RNI8-NT (a 0.1) and RNI8-NT-KAP (a = 0.1) models on 500 samples from CIFARIO dataset for varying amount of noise levels a.. Certified accuracy was estimated using the Monte Carlo procedure from (Cohen et al. 2019) using 100 samples for selection and 100000 samples for estimation.
Table A6: Comparison of adversarial accuracy against AutoAttack- with c 0.031 on CIFARIO for variations of the models with and without activation and input noise.
DATASET
MODEL
INPUT NOISE
ACTIVATION NOISE
AUTOATTACK, std
5.00
8.90
CIFAR10
RN18-NT (o - 0.1)
16.9
21.10
8.3
41.80
CO
RN 0.2, E (3)
57.30
60.70
19


Reviews:
[{"review_id": "p1XEJrYi2bf", "reviewer": "Reviewer_ckSs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles an important problem of the adversarial robustness of neural networks. Specifically, the authors of the paper propose a novel \"kernel average pooling\" layer that can be readily applied to most of the existing network architectures. Inspired by the recent success of neural network ensembles, the kernel average pooling layer applies a mean filter along the kernel dimension of the layer activation tensor. The authors of the paper also insights into why such a pooling layer can combat the problem of adversarial robustness, and conduct a series of experiments to demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "All in all, despite the interesting approach and observation of the proposed method, there are several limitations regarding the proposed approach. As such, I think the paper is marginally above the acceptance threshold.", "strength_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow. \n- The proposed method is technically sound and well motivated. \n- The experiments are carefully conducted. \n- The method seems to provide reasonable improvement in terms of adversarial robustness. \n\nWeaknesses: \n- Overall, I find this to be an interesting approach. The main complaint I have regarding this approach is that the robustness comes at the cost of significantly lower clean data accuracy, and at the same time, the improvement of adversarial robustness is not consistent, as seen from the fact that regular ResNet18 model with activation noise can sometimes outperform the proposed method in terms of adversarial robustness. \n- While there are some insights offered on why the proposed method can help adversarial robustness in section 3.4, most of the explanation is quite descriptive and lacks concrete details in my opinion. It might be good to go in-depth on this and offer some additional insights mathematically. \n- Maybe it makes sense to also benchmark the proposed method against some other cheap ensembling methods, such as [1]? \n- Other than adversarial robustness, does the proposed method offer other benefits? Does it help combat robustness against natural noise, model calibration, etc., since these are common benefits observed from a regular neural network ensemble? It might be interesting to investigate further other potential benefits as well.\n\n[1] Wen, Yeming, Dustin Tran, and Jimmy Ba. \"Batchensemble: an alternative approach to efficient ensemble and lifelong learning.\" arXiv preprint arXiv:2002.06715 (2020).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NwIEH7fxWFC", "reviewer": "Reviewer_6v33", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an architectural component called kernel average pooling which averages kernels (channels) of the input to the kernel average pooling layer.  They show that when using multiple KAP layers combined with random noise added to the input of each KAP layer, the adversarial robustness of the final model increases.", "summary_of_the_review": "Overall, I think that the authors should try using adaptive attacks against KAP and perform more ablations to better understand the impact of the noise in KAP and the impact of KAP layers on robustness.", "strength_and_weaknesses": "Strengths:\n- writing is clear and easy to understand\n- diagrams help with understanding KAP\n- experiments on a variety of datasets\n\nWeaknesses:\n- questionable robustness- Since this work proposes a randomized defense (without proposing or testing any adaptive attack against this defense) it is unclear whether KAP really provides robustness or not.\n- lack of ablation- the work immediately jumps into using multiple KAP layers with noise added before every layer.  What happens if you only add noise to the input?  Would using KAP improve on certified robustness via randomized smoothing in this case?  What about changing the number of KAP layers vs regular convolutional layers?\n- For CIFAR-10, I think the robust accuracy numbers for AT are a little low, probably due to suboptimal hyperparameters.  Could the authors use the training setup from Gowal et al. 2020?  I think that the choice in learning rate scheduler seems to change robustness numbers of AT by a lot.\nGowal, Sven, et al. \"Uncovering the limits of adversarial training against norm-bounded adversarial examples.\" arXiv preprint arXiv:2010.03593 (2020).\n- Since TRADES is also commonly used, it would also be good to use TRADES as a baseline too (using the training setup from Gowal et al 2020)\n- It would also be interesting to provide results on L2 robustness as well; after all the motivation of KAP was based on randomized smoothing which is a defense against L2 attacks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K_UyrX_pOX", "reviewer": "Reviewer_is9m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an algorithm to improve the adversarial robustness of neural networks based on kernel averaging and noise. They show that their method is competitive with adversarial training in some cases. However, the algorithm description is not clear enough and there are also not sufficient investigations on why the method works. \n", "summary_of_the_review": "The authors present a method for improving the robustness of neural networks based on kernel averaging and additive noise. Although there are some encouraging empirical results, it is difficult to understand why the method works. The authors should improve the presentation of the method, especially description of the algorithm, to improve the readability of the paper. The authors should also consider including ablation studies to help the readers understand what makes the method work. \n", "strength_and_weaknesses": "Strengths: \n- The authors combine averaging kernels with additive noise to improve the robustness of neural networks against adversarial attacks; some of these ingredients are known to help improve robustness.  \n\n- The method is competitive with state-of-art methods based on adversarial training on some datasets, without the use of adversarial training. \n\nWeaknesses: \n- Although there are some good empirical results, it is hard to understand and verify why the method works. Is it due to the use of noise for regularization in the network, or the use of averaging of kernels, or both? There is no ablation studies on these factors that might contribute to the robustness of the model, making the current study difficult to understand and reproduce. \n\n- The authors described 2D kernel averaging in Appendix A1. Yet unlike the spatial dimensions, there is no topological structure relating the different kernel dimensions. Why choose 2d instead of 3d or 1d? \n\n- While noise is known to improve robustness in previous work, it is not clear why kernel averaging helps improve robustness. The authors should consider further empirical/theoretical analysis to illustrate why it works, such as looking at its smoothing properties or effects on Lipschitz constants of the network. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2Kyn1kN3UyY", "reviewer": "Reviewer_nJ8u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes kernel average pooling operation to improve model robustness.", "summary_of_the_review": "This paper considers KAP operation to improve model robustness. It says that KAP focuses on learning feature ensembles that form local “committees” similar to those used in Boosting and Random Forests. However, I cannot see how KAP realizes feature ensembles. It seems to me that KAP is average pooling over the depth.", "strength_and_weaknesses": "Strength\n- The proposed operation is simple and economic.\n\nWeaknesses\n- The paper is not well-presented and well-written. \n   -  In the introduction section, it jumps direction into \"our central premise in this work\" to talk about ensemble without any hints or smooth transition.\n   - Especially, the section about kernel average pool is not clear and solid. Equation (3) does not make sense to me because $w_i.x$ for all $i$. It is hard to interpret this operation. The authors should give more context about the shape of $x$, $z$ and so on. In Algorithm 1, it seems that KAP is average pooling over the depth. So it is not novel to me. Additionally, after applying average pooling over the depth, the output tensor gets smaller, how you can get back the shape W,H,D as in the last line of Algorithm 1.\n- There are some vague arguments to me, for example \"individual kernels within a network are robust\". How can we justify if a kernel is robust?\n- How to interpret and understand Equation (9) because I cannot see any random factor in KAP.\n- The experiments are humble without comparing to other SOTA baselines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject