Learning the essential in less than 2k additional weights - a simple approach to improve image classification stability under corruptions
Authors: Kai Bäuerle
Patrick Müller
Syed Muhammad Kazim
Ivo Ihrke
Margret Keuper
Venue: Accepted by TMLR ()

Abstract: The performance of image classification on well-known benchmarks such as ImageNet is remarkable, but in safety-critical situations, the accuracy often drops significantly under adverse conditions. To counteract these performance drops, we propose a very simple modification to the models: we pre-pend a single, dimension preserving convolutional layer with a large linear kernel whose purpose it is to extract the information that is essential for image classification. We show that our simple modification can increase the robustness against common corruptions significantly, especially for corruptions of high severity. We demonstrate the impact of our channel-specific layers on ImageNet-100 and ImageNette classification tasks and show an increase of up to 30% accuracy on corrupted data in the top1 accuracy. Further, we conduct a set of designed experiments to qualify the conditions for our findings. Our main result is that a data- and network-dependent linear subspace carries the most important classification information (the essential), which our proposed pre-processing layer approximately identifies for most corruptions, and at very low cost.

Published in Transactions on Machine Learning Research (06/2024)
Learning the essential in less than 2k additional weights a simple approach to improve image classification stability under corruptions
Kai Bäuerle
kubbouerle@gmasl.com com
University of Monnheim, Germany
Patrick Müller
parrid.muellrrBuiiimmmmmmmm..a
University of Monnheim, Germany
Syed Muhammad Kazim
Syeel.kazzm@@uniiseeeee...
University of Siegen and Center for Sensor Systems (ZESS), Siegen
Ivo Ihrke
ivo. www.thkk@@niiseeenn...n
University of Siegen and Center for Sensor Systems (ZESS), Siegen
Margret Keuper
ceuper@uni-monnneee de
University of Monnheim and Mar Planck Institude for Informatics, Saarland Informatics Campus, Germany
Reviewed on OpenReview: https: oppnneevee.. net/ ftt//mrmm id- 42SuGwtIIN
Abstract
The performance of image classification well-known benchmarks such ImageNet is remarkable, but in safety-critical situations, the accuracy often drops significantly under adverse conditions. To counteract these performance drops, we propose a very simple mod- ification to the models: we pre-pend a single, dimension preserving convolutional layer with large lincar kernel whose purpose is to extract the information that is essential for image classification. We show that our simple modification can increase the robustness against common corruptions significantly, especially for corruptions of high severity. We demonstrate the impact of our channel-specific layers ImageNet-100 and ImageNette classification tasks and show an increase of up to 30% accuracy on corrupted data in the topl accuracy. Further, we conduct a set of designed experiments to qualify the conditions for our findings. Our main result is that data- and network-dependent lincar subspace carries the most important classification information (the essential), which our proposed pre-processing layer approximately identifies for most corruptions, and very low cost.
1 Introduction
Liu 2022d), improved for example by Neural Architecture Search (NAS) (Dosovitskiy et al. 2021 Tan & Le
Intensive research into DNN architectures
(H al. 2016
Szegedy et al. 2015 Tan & Le 2019
2019 and advanced training schemes Touvron et al.
2021
Chen et al. 2023b), has produced impressive
classification results (Dosovitskiy et al. 2021 Forct et al. 2021 The performance of models on well-known
benchnarks such as ImngeNNtt (Russakovske el al.] 2015). Cifar-100 (Krizhevsky & Hinton 20099 and others
has improved significantly over the last decade. However, a persistent challenge arises when these systems are exposed adverse conditions, such as changes in lighting, weather and other optical corruptions (Hendrycks
& Dietterich 2019 Müller et al., (023).. Despite achieving high accuracy on in-domain data, DNNs often experience a. significant drop in performance under these real-world challenges (üüller et al.
2023 Hendrycks

---

Published in Transactions on Machine Learning Research (06/2024)
et al.
2020
Sakaridis et al. 2021
Intensive research is therefore being carried out to increase model
robust ness to various disturbances.
A common strategy to increase robustness is to ap- propriately augment the original training dataset with relevant diversification. Such data augmen- tation techniques include geometric transforma- tions, cutouts and mixing of images (Yun et al. 2019 color jitter the simulation of out-of- distribution data by introducing common corrup- tions (Hendrycks & Dietterich 2019), and optical corruptions (Müller et al., 2023 Other methods involve adversarial training, whose objective yet at odds with robustness to some real-world corrup- tion types (Yin al.. 2019
Baseline
Ours
70
Accuracy (86
30
40
Arligmatiam
Cama
Delocus Blur Defocus sphenical
Trefai
Corruption Types
Figure 1: A pre-pended large kernel convolution layer (ours) can increase the robustness of classification net- works against unknown corruptions without additional data augmentation. Here, we show ResNet50 improved with trainable pre-pended filter evaluated on Im- ageNette (Howard] 2023) blur and corruptions from OpticsBench (Mäller et al. 2023). For each corrup- tion type, five levels of severity are shown from left to right. The variation, visualized via the box plots, results from five different seeds per model.
In this paper, propose very simple yet effective trick to improve model generalization, which con- sists of pre-pending to the model a single large kernel depth-wise convolution operation without strides. The proposed layer is trained with the model and can, in principle, learn a complete representation of the image (e.g. with identity mapping), but over-complete one. This is in contrast to the usual first model layers that create over-complete repre- sentations to facilitate sparse coding, Surprisingly, we find that our simple input layer trained solcly on clean data without particular augmentation strategies increases classification robustness to various corrup- tion types on multiple DNN architectures by up to 9.8% across 21 corruptions and by over 30% on specific severities with less than 2k additional parameters (e.g. only 0.008% for ResNet50). See Fig. 1 for an example. We thoroughly investigate this outcome with different methods to learn such large per-color-channel í.e. dimension preserving kernels, and compare to the respective baseline trained without the extra layer.
Major Findings and Contributions Our empirical study indicates that the proposed, dimension- preserving large kernel input layer, while being able to learn a complete data representation, tends to learn a subspace projection. As such, it extracts the crucial content from the input training samples, í.e. the es- sential, such as preserve a high accuracy on clean data. The dropping of non-essential parts of the signal, i.e. the learned subspace projection, automatically leads without requiring for any corruption specific data augmentation. We show this ImageNette (Howard 2023) and ImageNet-100 TTian et al., 2020) for the corruption types from Hendrycks & Dictterich (2019) and Müller et al. (2023 across diverse image classification models. Further, we explore in a signal theory- inspired study the properties of suitable kernels for the proposed layer, such as to gain deeper understanding
increase in the model's generalization ability
and foster future progress in this very cost efficient direction of improving model generalization.
2
Related Work
Model robustness and stability have been discussed under various perspectives. In the following, we first give brief overview standard benchmarks for the evaluation of classification robustness under corruptions, then, we summarize related work on model hardening through adversarial training. The proposed method differs significantly from these approaches, as no assumptions of potential threats to the model are made during training. Instead, our approach implicitly encourages the model to learn the relevant signal content while reducing parts of the input data that are less relevant (i.e. noise). To contextualize this finding, we also discuss prior work on the interplay between learned frequencies and model robustness, as well as prior art image resampling for neural networks.
2

---

Published in Transactions on Machine Learning Research (06/2024)
Image Corruptions and Data Augmentations To improve classification robustness against corruptions, data augmentation can be used to mimic the diversification of real data. AugMix (Hendrycks et al. 2020 improves robustness to common corruptions, Müller et al. (2023) use optical blur kernels to additionally improve accuracy on primary optical aberrations. Others use learned angmentation policies Cubuk ct al.
(2088) or add more abst.acc augmentations with image combinations (Yun et al. 2019 Berthelot el al.] 2019:
2020 feature map perturbations (Hendrycks et al., 2021,, combinations of fcature map perturbations and
image augmentation (Erichson et al., 20240, perturbed frequency representations (Yucel et al.
2023 -or by
Slapting/augmennti
the style of training images (Li et al. 2023
Hong et al., 2021 Xue et al.,
2023 Zhang
& Agrawala 2023 using SotA generative models (Ho et al. 2020 All these methods significantly increase
the training time of a model. The robustness they provide is limited to corruptions that are similar to the
augmented data. In comparison, our models are trained only on the clean dataset, avoiding i) the guessing of the corruptions, and ii) the computational overhead of an augmented data set, while offering improved gencralization ability of the trained model in many scttings.
Furthermore, Knowledge Dist (KD) can be used to dist.ill the robustness from a leacher model to
increase adversarial robustness (ooldluu et al. 2020 Zi et al. 2021: Huang et al. 2023 Zhao et al. 2022)
or out-of-distribution robustness (hou et al. 2023 The importance of data augmentation in KD training
is discussed in /Wang et al. 2022 Such approaches are expected to reach very high ranges of robustness, yet. they can only be applied when large pre-trained models are available for the considered domain.
Adversarial Attacks and Training Corruption benchmarks, e.g. Hendrycks & Dietterich (2019) Sakaridis et al. (2021). allow testing the model behavior w..tt. predetermined corruption types. In con- trast, adversarial attacks can add any (usually e-bounded) perturbation. They usually assume Lipschitz
continuity of robust models (Goodfellow et al., 2015 Kurakin et al. 2017 Wong et al.) 2020 Carlini & Wag-
ner
2017
Andriushchenko et al. 2020 [lyas et al. 2018).
When used during training, adversarial samples
can
be employed to harden a model [Goodfellow et al. 2015
Rony et al.
2019 Engstrom et al. 2019 Zhang
ct al.
2019 Wang ct al. 2020 Wu et al. 2020) Zhang et al.,
2019h, where some strategios involve additional
loss
terms (Engstrom el al.
2019 Zhang et al. 2019) or training data (Carmon el al. 2019 Sehwag et al.,
2021 Gowal et al.) 2021) e.g. IM extra samples generated by Ho et al.
(2020) are used in Gowal et al. ;; Rade & Moosavi- Deztooli 2021] Rebuffi et al. (2021) for adversarial training on CIFAR-10). While
[2021
significantly improving model robustness to adversarial samples, the additional training costs are immense.
The required compute resources increase e.g. by a factor of five to 15 even if the simple strategy of using adversarial samples during training is employed. Further, it has been discussed e.g. in Yin et al. (2019) Saikia et al. (0211; Gavrikov et al. (2023) that adversarial training is at odds with model generalization to some real real-world corruption types, which we focus on this work. In contrast. to the above discussed methods, our approach only requires a negligible overhead of less than 2k additional parameters and our models are trained using the respective standard training parameters, í.e. there are negligible extra-costs, while improving model robustness to various common corruptions.
Learned Frequencies and Robustness Previous works have studied the effect of learned frequencies within the shallow and deep layers of neural networks on model robustness (e.g. Yin et al. (019)). ). In Saikia et al. 2021), it is shown that regularizing a model to learn low frequencies and high frequencies separately can improve robustness to common corruptions. Grabinski et al. (022b) demonstrated a correlation between aliasing in CNN downsampling layers and their susceptibility to adversarial attacks. Several approaches re- duce or remove aliasing in the downsampling operators to improve the learned representations and their
robustness (Grabinski et al. 2022a Li et al. 2021 Zhang 2019 Hossain et al., 2023 Zou et al. 2020).
Further, Geirhos et al. (2018) showed that CNNs tend to focus on image textures rather than shapes to de-
termine an object class and Gavrikov et al. (023) discuss how adversarial training can shift this bias towards
shapes with both positive and negative effects on model robustness to common corruptions, depending on the corruption type. In contrast to these works, our approach solely focuses on the first model layer and no specific bias towards high or low frequencies is added. Yet. by providing only a single large kernel where the number of output channels equals the number of inputs, we implicitly encourage the model to focus on the essential part of the data.
Large Kernel CNNs in the "Era of ViTs" Vision Transformers Dosovitskly et al. 2021g (VTT) trained
on huge amounts of data have recently been outperforming classical small kernel CNNs (He et al. 2016
3

---

Published in Transactions on Machine Learning Research (06/2024)
To original CNN
Input Image HCi W)
Layor Output (C x
Trainable Kernel (C H2x XW)
The architecture of our proposed trainable input layer. We learn a single depth-wise convolutionn encourage the model to represent the essential, i.e. the part of the input data that crucial for classification.
Figure 2:
standard benchmarks, causing the community to shift its focus towards further improving transformer based architectures (Yu et al. 2022 Chen et al.) 2023a Zhai et al. 2022p. Contrarily, this has led to a trend towards increasing the filter size within deep layers of CNNs, e.g., it was shown that even kernels with 7x7 convolutions CNNs can allow them to outperform (Liu et al. 2022d self-attention based vision transformers (Dosovitskiy et al. 2021 Liu et al. 2022b). Extending on Tolstikhin et al. (202T). :
Smith et al. scale. Our approach is
(023) very recently show that CNNs perform on par with vision transformers therefore mainly evaluated CNNs, because they reach high accuracies even when trained rather low amounts of data, which facilitates our in-depth study. We show that our findings generalize to transformer models Dosoviiskiy ct al.
2021 Liu ct al. 2022b in Sec.
D.I in the appendix.
In Ding et al. 2022); Liu et al.
2022a Peng el al. 2017). the concept of large kernel CNNs is expanded with kernels up to sizes of 51x51 within the network, where handling the memory consumption is a challenge.
Global filter networks (Rao et al.] 2021) apply the filter in the frequency domain to allow for infinitely extended filters. These cases further highlight the benefit of using large kernels within the network. While we also using large kernel, our approach is different from the above: We use this filter only for the model input and perform information in the data, so that the entire model remains weight and can be trained with a low amount of data yet improves robustness.
mmage-to-imase mapping with it, encouraging the model to summarize the important
Learning CNN Inputs Several works have proposed to resample data in a non-uniform way at the model input or in deep layers allow for precise predictions in regions of interest, e.g. Ziwen et al. (2023 Tesse
al. 2023); Thavamani et al. 2021); Jin et al. 2022 Such approaches aim for individual downsampling patterns for every image. In contrast,
approach treats all images cqually and operates under the station- arily assumption, i.e. the applied convolution filter is constant over the entire image. Since downsampling can lead to aliasing, other works propose to learn to uniformly downsample so that more information is
preserved TTlleb & Milanfar 2021 Marin et al. 2019 Tu ct al. 2023 They all aim to improve the model's prediction accuracy. work for the downsampling task, where the output of the network restricted to be an image. Similarly, the output of our single layer is an image. Yet, our layer does not perform any resampling, avoiding potential aliasing, nor does it provide any non-linearity and can therefore be analyzed using lincar techniques.
particular, Talebi & Milanfar (2021) propose to optimize a small deep neural net-
3 Enhancing Prediction Stability with a Trainable Convolution Input Layer
Our aim is to investigate a simple approach to improve a model's stability under corruptions without in- creasing the training load. Our approach follows the intuition that it is beneficial to encourage a model to learn the relevant information from the data while neglccting superfluous parts of it, e.g. noise. To do so, we propose to add an extra convolution layer with a large kernel in front of the input layer of the model, where the output dimensions equal the input dimensions, i.c. no over-complete input representation
4

---

Published in Transactions on Machine Learning Research (06/2024)
be learned. If needed, the model can thus learn to preserve all data. Since, however, not all parts of the training data are valuable for the classification process (e.g. noise), we hypothesize that less important parts will likely be dropped in the learned mapping. The architecture, shown in Fig. 2 uses one kernel for each input channcl. The imput to the layer is the to-be-classfiied image with C' image color channels and a size of H x W. The kernel size of the convolutional input layer is C X HK x WK with a stride of 1, L.e. the result of the depth-wise convolution has the same dimensions, as the input image. In contrast to typical convolutional layer, this imput layer does not fuse the information of the color channels. The layer's output is propagated to the first layer of the otherwise unchanged DNN model, without additional non-linearity applied. Empirically as shown for example in Fig. 1 our simple approach leads to remarkable results.
Intuitively, the extra layer performs a specific linear transformation that can shift and/or block or emphasize the color-dependent content of an image. The data range of the kernel is not limited to positive numbers, so negative kernel values can also sharpen image content. This raises the question, which parts of the input data are preserved in our layer, f.e. whether the layer acts, for example, as an amplifier by spatially distributing the information in better way, whether acts as projection layer, where particular parts of the data are explicitly dropped. In the following, propose systematic approach to empirically test these options, by considering three different kernel classes.
Study Design The two key characteristies of a linear transformation are a) its rank and b) its conditioning, i.e. its noise amplification characteristics in the case a nominally full-rank transformation. The latter is characterized by the condition number (CN) of the transformation. Let ** g be the linear transformation effected by a convolution with the kernel g, and * *g- its inverse, then
CN(. + g) = AAmaxe
1/ 1/[Amin| CN((**
Aminl
1//Amax
where Amin and Amax are the minimum and the maximum eigenvalues of the transformation. The equation shows that the forward kernel and the inverse kernel have the same condition number, which explains our use of the term noise amplification. We use the condition number as a numerical indicator of the preservation of signal content
While the linear behavior of our proposed pre-processing layer is well understood, the reaction of the sub- sequent nonlinear network architecture to this modified input, is not. Our intention is to study the effect of the above-mentioned kernel properties on classification robustness.
We therefore introduce three kernel classes for further studying the properties of the proposed convolutional pre-processing layer, with the underlying hypothesis that signal content preservation or removal is decisive factor in the observed robustness increase.
name
CN
rank full full
class I (content preserving) unity/ medium large/infinite rank deficient
class 11 (fully trained/static)
class III (projection-type)
Table 1: Categories of kernels with their corresponding condition mumber (CN) and rank.
Class I: Content Preserving Kernels minimal noise amplifying kernel is one whose associated linear transformation has a determinant one: such transformation is unitary, i.e. vector norms are not changed.
For a convolution kernel, the associated matrix is a circulant matrix with the convolution kernel in the rows (assuming circular boundary conditions). The eigenvectors of circulant matrices form the Fourier basis, which is the underlying reason for the convolution theorem. The associated eigenvalucs coelficients. Since the product of the eigenvalues yields the determinant of the linear transformation, we see that all Fourier coefficients must have unit amplitude for the determinant to be unity. Since the Fourier coefficients are complex, they can still have arbitrary phases while fulfilling the unit amplitude constraint. An additional consideration, however, is that the associated kernels be real. This forces the constraint that
the Fourier
1 We emphasize that we not arguing in cenrmattion-thooeeiii but in numerical sense.
5

---

Published in Transactions on Machine Learning Research (06/2024)
([-w)=Gr(u) with G(w) being the Fourier spectrum, parameterized by angular frequency component w, of the spatial kernel function g(x). The constraint implies that we have N/2 real degrees of freedom to construct N pixel preseeving kernels, where N = Hx x WK. We use this parameterization for our practical layer implementation. A unit condition number can only be achieved for kernels of the same size as the image (225* x 225). We also experiment with smaller kernels (25 x 25) of the same construction, also referring to them as content preserving even though the condition number of the associated linear transformation is a low number above unity
are parameterized by their real value entries in the spatial domain. Positive and negative values are allowed, but not complex ones. In the absence of the aforementioned constraints of class I kernels, we are free to choose the size of the kernels. An ablation study of the kernel size for our proposed convolutional input layer is given in Sec. D.2 in the appendix, indicating that larger kernels can further improve model stability while trading-off accuracy on original data. We choose the kernel size be 25 x 25, which provides a favorable trade-off between both. We have observed that such freely trained kernels yield condition numbers in an intermediate range of 10 - 10 For comparison, we also include number of static kernels in class II.
Class I:: Fully Trained Kernels
Class III Projection-type Kernels have a CN that is (mumerically) infinite, since at least one Fourier coefficient is (numerically) zero. The removed subspace dimension is equivalent to the number of zero Fourier coefficients in the kernel's spectrum. We explore two ways to generate such kernels. First, we explore whether low-value Fourier coefficients in the fully learned kernels of class II can be replaced by zeros (thresholding). implying that the low values found by the optimization algorithm are effectively sufficient to remove the subspace in question from the data for all purposes of the nonlinear network part. Second, we encourage zero Fourier coefficients by means of an additional L1-regularization on the Fourier coefficients of the fully trained kernel. The associated sparsity then encourages projection-type kernels. The interesting characterizing number for projection- -tppe kernels is the dimensionality of the mull-space of the kernel, f.e. the number of its zero Fourier coefficienss. A larger number indicates a higher rate of signal content removal.
4 Experimental Evaluation
In the following, we evaluate the DNN prediction stability with our proposed, trainable image-to- image convolution input layer. The DNNs we evalu- ate are trained on different publicly available subsets of ImageNet (Russakovsky et al.) 2015) to allow for extensive experiments. ImageNette (Howard. 2023 is a datasct consisting of 10 ImageNNt classes. It has
Model
Version
CD
OB
CC
Net50
He et al.
2016
Base
#0.800
*0.592
*0.487
Res5N
He al.
Traimable
*0.775
*0.685
*0.55
AlexNet
R Krizevaay'
Base
*0,848
+0.572
*0.605
AlexNet
K Krizeeskyy
Trainable
*0.838
*0.670
*0660
Efficinn Net
Tan äz Le
2019
Base
0.907
0.604
0.605
Efficient
Tau & 1æ
2011
Trainable
0.903
0.629
0.633
MobileNer
Howard ec sl
Base
0.897
0,589
0.561
9,469 training and 3,925 validat ion images (Howard. 2023). ImageNet-100 (Tian al.,
MobileNet Howard el al.
Teaimbble
0.893
0.639
0.611
2020 uses 100
DeaseNett11 Huang et al.
Base
0.898
0.347
0.535
DensoNet161 Hnang al.
Trainabke
0.885
0.605
0.597
ImageNet classcs with total of 128k training and
XSEResNext.00
Base
0.936
0.652
0.607
5,000 validation images (Tian et al. 2020 In ad- dition to Tmage.Netee and ImageNNet-000 we also incorporate the full ImageNet-lk dataset, ensuring a comprehensive assessment of the effectiveness of our approach across varying scales and complexities (Soc. D.6..
XSEResNext50 He el al.
Trainable
0.933
0.677
0.659
NeXI Liu el al.
(2022
Base
0.824
0.516
0.489
ConyNeXt Lau ct al. 2122e
Trainable
0.796
0.565
0.539
ViT Dosovitskly et al 2121 Base
0.812
0.325
0.511
ViT Dosovitakty es ml 2021
Trainable
0.801
0.641
0.578
Swin v2
Liu el al. 2022b
Buse
*0.811
*0.571
*0.535
Swin v2
Lat ct a..
20226
Trainable
00.82
*0.698
*0.603
Table 2: Top1 Accuracy results on ImageNette for con-
All baseline models and all models with additional convolution input layer are trained from scratch on clean data, without additional data augmentation, L.e. following the standard training script. To en- sure a comparability between the baseline and our trained models, we only add our proposed input layer to the corresponding model and do not change any hyperparameters. The full details of chosen hy-
ventionally trained DNNs (Base) and additional fully trainable layer (Trainnale). CD = CCean Data, OB = OpticsBench (Müller et al. 2023) CC = Common corruptions (Hendrycks Dietterich 2019 * = aver- age from multiple seeds. The results on the corruption benchmarks are averaged across severity and corrup- tion. The highest accuracy per DNN is marked in bold.
6

---

Published in Transactions on Machine Learning Research (06/2024)
perparameters given in the supplementary material Sec. C The implementation is based on PyTorch and the training recipes follow maintainers & contributors (2016) In order to draw more comprehensive picture, some experiments were trained on five different seeds. Subsequently, all models are evaluated on
clean data and on two corrupted datasets (Hendrycks & Dietterich, 2019 Müller et al., 2023). The perfor-
mance of these trainings can be examined in Table 2 and 2
Furthermore, we trained our proposed model
adversarially and evaluate against the baseline in the supplementary material Soc.
D.7
original
defocus
astigmatism
coma
contrast
fog
Overview of different corruptions from OpticsBench's primary aberrations (astigmatism, coma) (Müller et al. 2023 and rotationally symmetric defocus blur, contrast and fog from (Hendrycks & Dictterich 2019h applied to an ImageNet sample. OpticsBench's astigmatism and coma introduce chro- matic aberration (visible at the flower petals) and directional blur, which can challenge DNNs differently than rotationally symmetric luminance blur. All corruptions are shown in the supplementary material, Sec.
Figure 3:
To test against corruptions, we use the common corruptions from Hendryeks Dictterich 2019 which are each binned into live dilierent severi- ties and apply them to ImageNette (Howard) 2023
ResNet50 AlexNet
MobileNecta DenseNet161
Swin VIT
ConvNext
and ImageNet-100 (Tian et al. 2020 To test for more diverse blur types, we also include the Op-
LicsBench from Müller et al. (2023 which covers primary optical aberrations and is similarly orga- nized. The blur kernels are size-matched to the besoocus-bur-ooe corrupiioo kernels from Hendryckss & Dietterich (009)) Fig. gives visual impression of some of the corruptions that evaluate
ACCous ACCquu
For the sake of readability, we summarize all dif- ferent corruptions into five super-categories (noise, blur, compression, weather, color) and take the av- erage of all the subcalegories. By calegorizing the corruptions, we can more effectively highlight the overall trends and impacts observed in our experi- ments. We provide the full details and figures with all individual corruptions in the supplementary ma- terial in Sec. D This section includes an in-depth breakdown each type of corruption with the dif- ferent severities.
-10
Noise
Blur
Compression Corruption Types
Wealter
Color
Figure 4: Relative topl accuracy improvements on ImageNette Howard 2023). A fully trainable input layer (class III can increase the robustness of classifi- cation networks against unknown corruptions without data augmentation. We evaluate different DNNs on blur and noise corruptions from OpticsBench (Müller et al. 2023 and Common Corruptions (Hendrycks & Dietterich 2019 For each corruption type, five levels of severity are shown from left to right.
4.1 Trainable Large Kernels can Improve Prediction Stability
In the following, we compare the prediction stability under corruption models with our proposed trainable convolutional pre-processing filter the respective basclines. First, the class I/ kernels are evaluated on the ImageNette corruptions. In Fig. g we plot the performance relative the baseline for better readability across different model families. Positive values indicate improvement the baseline model, and negative

---

Published in Transactions on Machine Learning Research (06/2024)
values indicate worse predictions. Absolute accuracies averaged over OpticsBench and Common Corruptions and on clean data are given in Table 2
Looking at all categories of corruption in Fig. it is noticeable that blur and noise benefit signif- icantly from the additional input filter compared to the baseline. For the other types, the accu- racies are on par with the bascline, and in some cases even get slightly below. This is especially true for color corruptions, which is to be expected since our proposed depth-wise convolutions can not learn color re-miiiinssooss Overall, yet in partic- ular for ResNet50,, the relative performance of the trainable input layer improves with increasing sever- ity for both noise and blur. As seen in Table - the prediction accuracy improves by large margin on OpticsBench (Müleer et al.) 2023) and Common Corruptions Hendrrykks & Dietterich 2019 for all models, e.g. 9.3% on OpticsBench and 7.8% on com- mon corruptions for ResNet50, while the accuracy on clean data is only slightly decreased. More re- cently published models, such as the Vision Trans- former (ViT) (Dosovitskiy ct al. 2021) and the Swin Transformer v2 (Liu et al. 2022b), outperform the baseline on corrupted images by a higher margin when combined with our trainable input layer. Swin v2 with the additional input filter increases the performance by 12.7% on OpticsBench and 6.8% on Common Corruptions, while only slightly lacking accuracy on the clean data of about 0.9%. A more detailed eval- uation of these models can be found in the supplementary material Sec. D. A model, which is specially designed for small ImtgeNet subsets, such as the XSEResNext50 (Howard, 2023), is also able to improve its performance on corrupted images despite its high bascline accuracy, suggesting that incorporating the latest training techniques to improve model performance does not undercut the benefits of the proposed method.
Model
Version
CD
OB
CC
ResNet50
Base
*0.801
*0.536
*0.406
ResNet50
Trainable
*0.797
*0.558
*0.437
AlexNet
Base
0.698
0.339
0.307
AlexNet
Trainable
0.671
0.363
0.344
EfficientNet
Base
0.796
0.480
0.394
EfficientNet
Trainable
0.795
0.509
0.393
MobileNet
Base
0.780
0.470
0.344
MobileNet
Trainable
0.761
0.501
0.404
ViT ViT
Base
0.684
0.396
0.296
Trainable
0.677
0.437
0.331
Swin v2 (tiny)
Base
0.779
0.433
0.323
Swin v2 (tiny)
Trainable
0.774
0.476
0.379
Table 3: Top1 accuracy results on ImageNet-100 for conventionally trained DNNs (Base) and additional fully trainable layer (Trainable). CD= Clean Data,
OB OptiesBench (Mäller al. 2023 CC = Com- mon corruptions (Hendrycks & Dietterich 2019). multiple seeds. The results onOB and CC are aver- aged across severity and corruption.
Second, the same experiment is performed on ImageNet-1100 with the same types of corruption. Table 3 lists the achieved accuracies on clean data,
Model
Version
CD
OB
CC
Res.
Base
0.781
0.482
0.393
ResNet50
Fully trainable
0.774
0.509
0.425
OpticsBench (Müller et al., 2023) and common cor-
Swin v2 (tiny)
Base
0.788
0.395
0.292
ruptions (enddrykk & Dietterich 2019). The train- able layer improves again on average over the cor- ruptions for each DNN. However, compared to the results on ImageNette, the improvements are now smaller. ResNet550 with the trainable large input kernel performs best with an increase in accuracy of +2.2% on OpticsBench and +3.1% on common cor- ruptions. More results on ImageNet-100 are given in the supplementary material Sec. D.6
Swin v2 (tiny)
Fully trainable
0.771
0.426
0.326
Swin v2 (base)
Base
0.783
0.423
0.320
Swin v2 (base)
Fully trainable
0.783
0.459
0.355
Table 4: Results ImageNet-1k for ResNet50, MobileNet v3 large, and Swin Transformer v2 (tiny and base). CD=Clean Data, OB = OpticsBench, CC=Common Corruptions. The results on OB and CC are averaged across severity and corruption.
On the ImageNet-1k datasct (Russakovsky et al. 2015 the performance differences between the baselines and our proposed input layer models are slightly lower than on ImageNet-100. However, for higher severities our models significantly outperform the baseline. Especially the transformer-baseed models benefit from the new input layer. These effects are visualized in the supplementary material Fig. to 50 The performance on clean data is just slightly in favor of the baseline while evaluating on larger datasets as in Table E The larger version of the transformer-basee Swin v2 is even on par with the bascline the clean data, while outperforming the bascline on both corrupted evaluation datasets by 3.6% and 3.5%.
8

---

Published in Transactions on Machine Learning Research (06/2024)
In the following, we investigate with help of our proposed kernel classes I and III which properties the learned kernels have. Understanding these properties is particularly intriguing because they allow the kernels to represent the input data without an increase in dimension, while the subsequent networks generalize better to many unseen corruption types.
4.2 Which Properties of Trainable Kernels can help? Comparing Kernel Classes
The results in Fig. Table and X show an im-
provement in prediction over many corruption types. This raises the question of what kernel properties cause these results, and whether they can be im- proved further with different kernels. In order to deepen the considerations from Sec. g we analyze numerous variants of the input layer kernel for a
Dase
Freserve content (cless
prior Icless III
Prevervw condent large (cless
Fully trunnble [class
60
Accuracy [%]]
ResNet50 and compare the different kernel classes I-III. The results for more models are given in the
supplementary material Sec. D Table 5 lists the ab- solute accuracies.
30
We visualize in Fig. 5 the results on ImageNette for ResNet50 in an absolute fashion to compare the im- pact of the different kernel classes on corruption ac- curacy. We plot the baseline and the fully trainable layer (class II) together with class and III kernel models. The content preserving kernels represent class I and the L1 prior represents class III. Except for color corruptions, both the L1 prior model and the fully trainable kernel model help to stabilize the predictions of the baseline model, while the content preserving model helps only for noise and blur at high severities.
Noise
Blur
Compression Corruption Types
Weather
Color
Figure 5: Comparison of different kernel types on cor- rupted ImageNette data for ResNet50. For additional kernels, see Table 5
Interestingly, the L1 prior model produces the most favorable results for noise and blur corruptions, fol- lowed by the fully trainable kernel, and achieves an accuracy gain of more than 40% at noise severi- ties 4 and 5. The predictions for blur with the L1 prior model remain almost constant at 72% accu- racy, while the baseline accuracy drops below 40% with increasing severity. The fully trainable kernel model largely stabilizes the predictions, but drops by around 10% at higher severities compared to the L1 prior model. The compression corruption type is more challenging for all models compared to the baseline, while the fully trainable kernel and the L1 prior models perform similarly and increase in accu- racy from severity 3. Interestingly, the baseline also performs quite well here, which may be due to com- pression artifacts within the original training data.
Kernel type
class
CD
OB
CC
None (Base)
*0.800
*0.592
*0.487
Preserve content
0.754
0.476
0.513
Preserve content large
0.645
0.478
0.440
Conv2D KS-25
II
0.635
0.601
0.501
Fully trainable
II
*0.775
*0.685
*0.565
Random initialization
0.711
0.673
0.550
Ll prior
III
0.712
0.699
0.567
Directional blur filter Gauss blur filter
II 0.768
0.437
0.345
II 0.764
0.668
0.541
Table 5: Topl accuracy results on ImageNette for ResNet50 and different input layer large kernel types.
CD = Clean Data, OB OpticsBench (Müller al.)
2023 CC Common corruptions (Hendrycks Di- etterich 2019).
average from multiple seeds. The results the two corruption benchmarks are averaged across severity and corruption. Bold: best model, un- derlinc: second best.
The content preserving kernel model performs sig- nificantly worse than the baseline, while the L1 prior and the fully trainable kernels tend to slightly increase prediction stability towards higher severities even for hard corruptions (e.g. weather). While the L1 kernel yields rather high robustness, it is significantly underperforming on clean data.
In summary, the L1 prior and the fully trainable versions seem to follow a similar pattern at similar levels of corruption severities in Fig. 3 and largely increase the accuracy compared the baseline for most corruptions and severities. Yet, clean data. (e.g. in Table 5 the L1 prior leads to significantly lower accuracy (-6.%%
9

---

Published in Transactions on Machine Learning Research (06/2024)
w.r.t. the fully trainable filter). In contrast, the content preserving kernel model in Fig. 5 follows similar trend to the baseline for noise and blur and has the worst performance for all corruptions.
This analysis indicates that for many corruption types, it is beneficial to explicitly encourage the convolu- tional image-to-image layer to project the input image onto subspace in which certain spatial frequencies are not represented. The un-regularized fully trainable layer seems to do this implicitly, while preserving the essential information such as to perform well on clean data.
To further deepen our understanding, we present more experiments with other kernels from the three classes in addition to the kernel variants shown in Fig, S First, different sizes of class / kernels are compared to see the trade-off between content-preed settent-preeerattoon and feature locality. The subsequent group analyses different aspects of trainable class I/ filters. The last group compares different. class III filters to further study the assumed subspace projection.
Content Preserving Kernels (Class I): To have fully content preserving filter, the kernel needs to have the same size the to-be-convolved images. Thus, experiment with two sizes of class I kernels. The first kernel has the same size as the imput images (225 x 225). To also be able to compare the class I filters with other classes, we designed and trained a 25 x 25 'content preserving" kernel, i.e. a filter that would be content preserving for 25 x 25 patches. The accuracy of both kernels is given Table The larger kernel performs siguificantly worse on the clean data as well as on the common corruptions. Moreover, in the OpticsBench dataset and therefore also in the blur corruptions in Fig. both kernel sizes perform comparably poorly, Filters that purely re-arrange content, whether they preserve locality not, do not lead to an increase in prediction stability under corruptions.
Fully Trainable Kernels (Class II): We perform two additional experiments: one which replaces our trainable color-dependent (depth-wise) convolution layer with a standard convolution layer the same kernel size (25 x 25 x 3). The other uses the proposed depth-wise convolution layer, but with random kernel initial- ization, which validates the benefit of our fully trainable kernel. Both kernels are class I7 representations.
The standard convolution layer achieves signifi- cantly lower accuracy on clean data and both bench- marks. Re-combining color channels provides the pre-filtering with more capacity and the ability to better overfit the training data, i.e. it provides less generalization. Yet, yields better results than the class content preserving kernel model.
o.
sands
mequincc 20
mequaaec 4 db
The random initialization model tests initialization of the kernel and achieves comparable results to the fully trainable case for the corrupted data while suf- fering some loss in peak performance on clean data.
Spachy 80
60
(a) Without Regularization
(b) L1 Regularization
Figure 6: Evolution of spectra of (a) Fully Trainable and (b) L1 Prior kernels The bar hoight indicates the average of the absolute value the Fourier coefficients in different frequency bands (DC component the front). Each epoch is normalized separately.
To test the impact of the trainability of our model, we also evaluate two non-trainable blur filters, one rotational-symmetric Gauss blur filter and direc- tional (horizontal coma) blur filter obtained from OptiesBench (Miller et al. 2023) These have both lowpass characteristics and remove high frequency con- tent. From the results in Table the two blur filters have an in-domain accuracy comparable to the fully trainable model. Only the Gaussian blur performs better than the baseline on the two corruption bench- marks, while the directional blur model performs worse. Both perform substantially worse than the fully trained filter.
Projection-type Kernels (Class III): The class III kernel (L1 prior) in Table 5 tends to reduce the frequencies in the trainable layer via Lasso regression on the frequency domain of the kernel. This can also be visualized in Fig. g (b) where the kernel learns to discard higher frequencies. From Fig. t (a), it is evident that the optimization reduces the Fourier coefficients of higher frequencies of freely learned class I/ kernels as well. To check whether numerically small frequency coefficients contribute to the networks' outputs, we
10

---

Published in Transactions on Machine Learning Research (06/2024)
use our trained models and remove the low magnitude frequencies without retraining. With this approach, we were able to show that even after removing over 80 % of the frequency information (not necessarily high-frequencies), the model's performance is stable. The results over multiple removal intervals with our models can be examined in Fig. i
With the insights of Fig. and the increments performance while training with an L1 prior on the frequency domain of the trainable kernel, we conclude that low values in the fully trainable class II kernels are numerically zero, i.e. they effectively implement projection-type kernels.
4.3 Comparison with Augmentation and Joint Trainable Large Kernel and Augmentation
Data augmentation is the de-facto standard for ro- bustifying deep learning models. It comes at the cost of a) an increased training data set and there- fore enhanced training times, and b) a model for the expected corruptions must be known or guessod.
MobilelenVV large
666
Since our proposed technique does not rely on these prerequisites, study how the two approaches compare and whether their combination can yield further benefits.
56
10
20
30 40 50 60 70
80
90
Frequencies Removed [36]
We trained multiple models with Aug- Mix [Hendrycks al., 20200, generic data augmentation method, with default settings and compare ResNet50 models trained without AugMix (Base), with the same model trained with our proposed input layer, but without Augmix (Trainable), and a model with AugMix in combination with our proposed input layer (Augmix & Trainable). The trained models were evaluated on clean data, OpticsBench Miüler et al. 2023) corrupted data and Common Corruptions (Hendrycks Dictterich 2019 The result of these experiments is shown in Table C
Figure 7: Thresholding the trained convolutional preprocessing kernels in the frequency domain only marginally decreases the accuracy on Optics- Bench (Müller et al. 2023
As AugMix works with similar corruption meth- ods, in Common Corruptions, the results are improved by only using AugMix on both datasets. When trained on ImageNette, using our pro-
Dataset
Version
CD
OB
Base
*0.800
+0.592
$0.487
ImageNette
Trainable (ours)
*0.775
*0.685
*0.565
AngMix
0.781
0.561
0.512
AugMix & Trainabee fours)
0.795
0.774
0.639
posed imput layer (Trainable), the improvement outperforms a pure AugMix setting, while on ImageNet-100, the converse is true. However, with either dataset, using AugMix in combination with our proposed input layer (AugMix & Trainable), the performance in both corrupted datasets increases significantly over every other combination. On Imagenet-100 the combination of AugMix and our imput layer even outperforms the baseline on clean data.
Base
*0.801
*0.33
"0.000
ImageNet-100
Trainable (ours)
*0.797
40.558
*0.437
AngMix
0.509
0.639
0.518
AugMix Trainable (ours)
0.814
0.663
0.533
Table 6:
Results from AugMix (Hendryeks et al.
2020
training experiments on ImageNette (Howard,
2023
and ImageNet-100 (Tian et al. 2020) datascts. (Müller al.) CC Common corruptions (Hendrycks Di- etterich 2019). multiple seeds.
Clean Data, OB = Optics
2023
Table presents comprchensive comparison of various robustness methods applied to ImageNet- 1k using ResNet50. Next to the accuracy different datasets Clean Data (CD), OpticsBench (OB), and Common Corruptions (CC)- we compare the costs per epoch (CpE). Our proposed input layer model demonstrates a notable improvement in OB and CC (0.509 and 0.425, respectively) with only a slight increase in CpE. When combined with Deepengment (DA), our method (DA & fully trainable) further enhances performance on OB and CC (0.641 and 0.542, respectively) while maintaining a competitive CpE of 5,083 seconds. Furthermore, this showcases, the ability to combine our proposed input layer with state-of theth--attttttt robustness methods, such Augmentment Hendrycks et al. 2021
11

---

Published in Transactions on Machine Learning Research (06/2024)
5
Discussion
Our main findings can be summarized as follows: simple convolutional pre-processing layer can signif- icanlly improve the robustness against unseen cor- ruptions even when trained only on clean data with- out dedicated augmentation schemes. An analysis of the learned kernels and experiments with different classes of kernels that were designed to explore dif- ferent levels of preservation of signal content show that projection-type kernels lead to the most ro- bust results in the majority of corruptions while not significantly reducing peak performance and perfor- mance in the case of difficult corruptions (weather and color).
Version
CD
OB
CC
CpE [s]
Base
0.781
0.182
0.393
1,510
Fully trainable (ours)
0.774
0.509
0.426
1,580
AngMix [euddryls es al.
2020
0.773
0.633
0.511
11,338
DeepAngrennt Henndydde
4020
0.769
0.63T
0.529
4,971
DA & Fully Trainale [ours
0.776
0.641
0.542
5,083
NoisyMix
Eriehson
2024
0.709
0.607
0.532
7,958
SIN IN
Geirbos
0.750
0.537
0.457
3,059
DAD
Zikou e6 aL.
0.802
0.502
0.195
10,352
Table 7: Comparisons of different robustness methods on ImageNet-Ik with ResNet50. CD=Clean Data, OB OpticsBench, CC=Commor Corruptions, CpE' Costs per epoch in seconds, DA = DeepAugment. The results on OB and CC are averaged across severity and corruption.
This indicates that a removal of signal content can aid the robustness of classification networks against unknown corruptions with the associated benefits of 1) not having to model expected corruptions for augmentation-styyl training and 2) computationally favorable implementation: only x 2000 additional coefficients are needed and training can be performed on a smaller dataset as compared to an augmentation approach. Besides this, several methods exist, which increase robustness by removing signal content: dropping high frequency wavelet coefficients (Li et al., 2021) allows for robust high-level features. Discarding high frequency content also helps in gaining robustness to common corruptions Grabinski et al. (2022a); Zhang (2019). However, our prepended layer does add negligible ext extra-costs without having to transform lo any co-domain such as a Wavelet Fourier basis.
We observe the usual trade-off of peak-performance vs. robustness. Our experiments indicate that enforcing sparsity of the frequency content of the proposed convolutional pre-processing layer is an alternative way of achieving robust classification results, which is in line with previous findings (Yin el al. 2019 While this trade-off also indicates that the signal content responsible for successful classification and possible corruptions do not occupy entirely disparate linear subspaces, it appears as if the signal content responsible for successful classification is essentially a linear subspace of the data. Our proposed convolutional pre- pre-processing layer can therefore be interpreted as being an approximation of the responsible linear subspace. Yet, forcing the projection with the sparsity prior on frequencies yields shift in the tradc-off from high peak-performance to higher robustness. In contrast, content without any prior while better preserving the performance on clean data. A discussion of the relationship to sparse coding is given in the supplementary matcrial.
rained non-overcomplete layer can learn to represent the essential
The complementary experiment of designing signal-contenl preserving kernels yielded no appreciable perfor- mance improvement over the baseline, where the baseline, being an identity transformation, can be inter- preted as signal-content preserving as well. This is an indication that the mull-space of the high-performance kernels carries information that can lead to over-fitting with associated decrease in robustness of the classification model.
Our proposed layer could, in principle learn a complete representation of the input images (an identity map- ping) without increasing the loss in clean accuracy. The remaining question is why the layer actually learns the subspace projection that allows to focus on more essential information and leads to better generalization. One reason could be that the sparse classes, that are the output of classification, can propagate towards the input layer, paired with the spatial inductive bias of the convolution operation itself, which is trained on image data that is heavily correlated, i.e. neighboring pixels tend to be similar. The layer might therefore be biased towards the global image structures and learn to represent fine details only where they are needed to perform well on the training data (e.g. the essential high frequency details), which yields the observed ben- efits. It would be interesting to study why this is not occurring in the standard initial layers of the network, without additional regularization (Yin et al. 2019). We hypothesize that this is due to the over-completencs of the representations of most carly layers, making it quite likely for a model to represent relevant content well noise. In contrast, our simple dimeesioon-preervving layer can at most preserve the input data, and
12

---

Published in Transactions on Machine Learning Research (06/2024)
needs gradient signal from the model loss to learn to do so. It therefore learns to predominantly represent the signal that is needed for the task at hand, i.e. the essential.
6 Conclusion & Future Work
We describe novel, very simple robustifying scheme for classification networks that has the attractive features of being light-weight, both in training and in inference mode, and not requiring knowledge on the corruption model. Furthermore, we showed that the model is compatible with image augmentation. We therefore believe that this simple technique has a. large application potential. However, the current paper is only a first step into its analysis. The convolution proposed and studied in this paper is a space-invariant linear transformation, which appears like a sensible choice for classification problems. An open question is whether other problems like semantic segmentation, tracking, etc. can benefit from similar strategies. In this context, it is further unclear whether space-invariance is desired property or whether more general lincar transformations tions could be beneficial, e.g. in yielding closer approximations to the relevant subspace of the signal content. connected question is how space-variant processing by the follow-up network affects the initially space- invariant processing by our proposed layer.
Acknowledgments
Syed Muhammad Kazim, Ivo Thrke and Margret Keuper acknowledge funding from the DFG research unit DFG-FOR 5336 "Learning to Sense".
References
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. CoRR, abs/ abs/1992.00009, 2019. URL http:
//arxiv.org/ab//10. 00049
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Proceedings of the Europeon Conference on Computer Vision (ECCV) 2020, 2020.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation auchoring. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Prinacy (SP), pp. 39-57. IEEE, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Tiang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, volume 32, 2019.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Gryener, Basil Mustafa, Lueas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Bureu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language- image model, 2023a.
Xiangning Chen, Chen Liang, Da Huang, Estoban Roal, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, and Quoe V. Le. Symbolic discovery optimization algorithms, 2023b.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Francesco Croce and Matthias Hcin. Reliable evaluation of rob with an ensemble of diverse parameter-free attacks. CoRR, abs/2003.01690, 2020. URL. https://arxiv http:/iwwwrraab//////ea/ 0000/
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXic preprint arXiv: 1ew:1805.0900 09501, 2018.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding, Scaling up your kernels to 31x31: Revisiting large kernel design in enns. In Proceediigg of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 11963-11977 June 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of the International Conference Learning Represenations (ICLR), 2021. doi: 10.48550/arXii 20t.i.v000011199))
Logan Engstrom, Andrew Nyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL
Benjamin Erichson, Soon Hoe Lim, Winnie Xu, Francisco Utrera, Ziang Cao, and Michael Mahoney, NoisyMix: Boosting model robustness to common corruptions. In Proceedings of The 27th Interna- lional Conference on Arlificial Intelligence and Statislics, pp. 4033 4441.. PMLR, 2024. URL https:
//proceedings.nt mlr. ptt.s..rrss//vee///// erichson24a. html ISSN: 2640-3498.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2021.
Paul Gavrikov, Janis Keuper, and Margret Keuper. An extended study of human-like behavior under adver- sarial training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2361-2368, June 2023.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Micah Goldblum, Liam Fowl, Sohcil Feizi, and Tom Goldstein. Adversarially robust distillation. 34(4): 3996 4003, 2020. ISSN 2374-3468. doi: Number: 04.
URL https://ojs..... aaa:: org/index.
pttp/aatt/artiice/rrieee////
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.
In Proceedings of the International Conference on Learning Represenlations (ICLR), 2015.
Sven Gowal, Sylvestre-Alviss Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advonces in Neural Information Processing Systems, 34, 2021.
Julia Grabinski, Steflen Jung, Janis Keuper, and Margret Keuper. Prequencyloweut pooling plug & play against catastrophic overfitting. In Proceedings of the European Conference on Computer Vision (ECCV), 2022a.
Julia Grabinski, Janis Keuper, and Margret Keuper. Aliasing and adversarial robust, generalization of cmns. Machine Learning, pp. 1-27, 2022b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi:
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. tricks for image classification with convolutional neural networks. https::/openaccees. thecvf www.totteetttcccoeooooocccccc.tc / Sattee_ag__o_ of_ Tww_kric_______ fae__fo__mmaaeag Image_ Classification with Convolutional Neuriaaooaaeuuuullla M Networks CVPR CVPR_2013 2019 __ae___pp__________________ paper.html .htll
Bag of URL
pp. 558-567,
14

---

Published in Transactions on Machine Learning Research (06/2024)
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In Proceedings of the International Conference on Learning Represenations (TCLR), 2019.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty, In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributiin generalization. In Proceedings of the IEEE/CVI International Conference on Computer Vision, pp. 8340 8349, 2021.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Contont-adaptive downsampling in convolutional neural networks. In IEEE/CVF Conference Computer Vision and Pallern Recognition Workshops (CVPRW), The 6th Efficient Deep Learning for Computer Vision (ECV) Workshop, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. Advances in Neural Information Processing Systems, 33:6840-685/ 2020.
Minui Hong, Jinwoo Choi, and Gunhee Kim. StyleMix: Separating content and style for enhanced data augmentation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Md Tahmid Hossain, Shyh Wei Teng, Guojun Lu, Mohammad Arifur Rahman, and Ferdous Sohel. Anti- aliasing deep image classifiers using novel depth adaptive blurring and activation function. Newrocompuling, 536:164-174, 2023.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetio, and Hartwig Adam. MobileNets: Elficient convolutional neural networks for mobile vision applications. URL
04861
Jeremy Howard. Imagenette.
2023.
Bo Huang, Mingyang Chen, Yi Wang, Junda Lu, Minhao Cheng, and Wei Wang. Boosting accuracy and robustness of student models via adaptive adversarial distillation. In 2023 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 24668 24677. IEEE, 2023. ISBN 9798350301298, doi: N0.1109/CVPR52229.222 202.202223.3333 URL htttps://eeeeppleeeee leee org//accumet/12231377//
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connocted convo- lutional networks. pp. 4700-4708. URL https://openaccess: ctacccccssssstteteeeeeeee thectheeeeccce www//ooteet__crrnocccccc 1ooontent_crpr_2217/aaaaa Huang Densely Connected Con.eleoiooooooooooooll CVPR, 2017 2017/_aper.htt. html
Andrew Tyass, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning (ICML), July 2018.
Chen Jin, Ryutaro Tanno, Thomy Mertzanidou, Eleftheria Panagiotaki, and Daniel C. Alexander. Learning to downsample for segmentation of ultra-high resolution images. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data aug- mentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pottern Recognition, pp. 18963-18974, 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep
convolutional neural networks. In Advances in Newral Information Processing Systems, vol-
wne 25. Curran Associates, Inc. URL https://papers. copps.rr wiips.cc/paprr.fiiee//pppppp filess/paper/2022/aaas
33982239b667666447744446666 www.lsbt-bbtrat..... html
Alexcy Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint 1XXiv1611.01220 2017.
Qiufu Li, Linlin Shen, Sheng Guo, and Zhihui Lai. Wavecnet: Wavelet integrated cmns to suppress aliasing effect for noise-robust image classification. IEEE Transactions on Image Processing, 30:7074 7089,, 2021. ISSN 1941-0042. doi: 111109/tip.2221.311111
Yumeng Li, Dan Zhang, Margret Keuper, and Anna Khoreva. Intra-& extra-source exemplar-based style synthesis for improved domain generalization. IJCV, 2023.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. Xiv preprint arXio:2207.0622, 2022a.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In International Conference on Compuler Vision and Pullern Recognition (CVPR), 2022b.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A con- vnet for the 2020s. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022c.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11976-11986, 2022d.
TorchVision maintainers and contributors. TorchVision: PyTorch's Computer Vision library. https://
rithib..com/pptorccc///iiilll November 2016.
Dmitrii Marin, Zijian He, Peter Vajda, Priyam Chatterjee, Sam Tsai, Fei Yang, and Yuri Boykov. Effi cient segmentation: Learning downsampling near semantic boundaries. In Proceedings of the IEEE/CVP International Conference on Computer Vision (ICCV), October 2019.
Patrick Müller, Alexander Braun, and Margret Keuper. Classification robustness common optical aber- rations. In Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops (TC- CVW), pp. 3632 3643, 2023.
Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters improve semantic segmentation by global convolutional network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Rahul Rade and Scyed-Mohsen Hoosavi-Dezfool Helper-based adversarial training: Reducing excessive margin to achieve better accuracy vs. robustness trade-off. Tn ICML 2021 Workshop on Adversarial Machine Learning, 2021.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021.
Sylvestre- Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Fixing data augmentation to improve adversarial robustness, 2021.
Jérôme Rony, Luiz G Hafemann, Luiz Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. De- coupling direction and norm for efficient gradient-basse 12 adversarial attacks and defenses. In Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4322-4333 4330, 2019.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 252, 2015. ISSN
0920-5691, 1573-1405.
doi:
Tonmoy Saikia, Cordelia Schmid, and Thomas Brox. Improving robustness against common corruptions with frequency biased models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10211-10220, October 2021.
Christos Sakaridis, Dengxin Dai, and Luc Van Gool. ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding. In 2021 IEEE/CVF International Con- ference on Computer Vision (ICCV) pp. 10745-10755. IEEE, 2021. ISBN 978-1-66542-812-5. doi: /TA.////CCVV49222221...111
Vikash Schwag, Saced Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Improving adversarial robustness using proxy distributions, 2021.
Samuel L. Smith, Andrew Brock, Leonard Berrada, and Soham De. ConvNets match vision transformers at scale, 2023.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deoper with convolutions. In 2015 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 1 1-9, 2015. doi: 10.1109/CVPR... 201.C
ISSN: 1063-6919.
Hossein Talebi and Peyman Milanfar. Learning to resize images for computer vision tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 497-506, October 2021.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference Machine Learning, pp. 6105-6114. PMLR, 2019. ISSN: 2640-3498.
Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan. Fovea: Foveated image magni- fication for autonomous navigation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15519-15528, 2021. doi: 10.1109 AC922C9922222222
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, Augusl 23 28, 2020, Proceedings, Part XI, pp. 776-794.
2020. ISBN 978-3-030-58620-1. doi: 00.10077978--- 0110//9788------000000 3-030-58621-8 45.
lyy O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, and Alexey Dosovitskiy. MLP-mixer: An all-MLP architecture for vision. In Advances in Neural Information Processing Systems, volume 34, pp. 24261-24272. Curran Associates, Inc., 2021.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 32 12-42, October 2021.
Zhengzhong Tu, Peyman Milanfar, and Hossein Talebi. Muller: Multilayer laplacian resizer for vision. Ar abs abs/2330 Abs/2304.02859, 2023.
Huan Wang, Suhas Lohit, Michael N. Jones, and Yun Fu. What makes a 'good' data augmentation in knowl- edge distillation a statistical perspective. 35:13456- 13469, 2022. URL /ttp://///htss//pppoeeeeneeerrrr
html
Yisen Wang, Difan Zou, Jinfong Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Represen- lations, 2020.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Represenlations, 2020.
Dongxian Wu, Shu-Tao Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Aduances in Newral Information Processing Systems, 33:2958 2969, 2020.
Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. Tn CVPR, 2023.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Proceedings of the 33rd International Conference on Newral Information Processing Systems, pp. 13276 13286, Red Hook, NY, USA, 2019. Curran Associates Inc.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022.
Mehmet Kerim Yucel, Ramazan Gokberk Cinbis, and Pinar Duygulu. Hybridaugmentt++:: Unified frequency spectra perturbations for model robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5718-5728, October 2023.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. CutMix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 6022-6031. IEEE, 2019. ISBN 978 978-1-72814-803-8 doi:
10.1109/TCCV. 2019.00612.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12104-12113, June 2022.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. The- oretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, 2019.
Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Tn ICCV, 2023.
Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pp. 7324-7334. PMLR, 2019.
Shiji Zhao, Jie Yu, Zhenlong Sun, Bo Zhang, and Xingxing Wei. Enhanced accuracy and robustness via multi- teacher adversarial distillation. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 pp. 585 602. Springer Nature Switzer- land, 2022. ISBN 978-3-031-19772-7. doi: 0011007/978-33-311-19777
Andy Zhou, Jindong Wang, Yu-Xiong Wang, and Haohan Wang. Distilling out-of-distributtion robustness from vision-language foundation models. 2023. URL
Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. In 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 16423 16432. IEEE, 2021. ISBN 978-1-66542-812.-1 doi: 10.1109/1CCV44892222221.. URL https: feee. org.//ocmeent//..08555
Chen Ziwen, Kaushik Patnaik, Shuangfei Zhai, Alvin Wan, Zhile Ren, Alex Schwing, Alex Colburn, and Li Fuxin. Autofocusformer: Image segmentation off the grid. In CVPR, 2023.
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. In BMVC, 2020.
18


Reviews:
[{"review_id": "4N3vy05eyd", "reviewer": "Reviewer_5xbk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ the method is very interesting, adding a simple and efficient component that can improve the performances. \n\n+ the empirical improvement is notable, over the two datasets reported. \n\n+ there are discussions of comparisons of kernels to use. \n\nWeakness:\n- the biggest weakness would be the empirical scope. Machine learning robustness has been studied extensively with tons of methods, tested on much bigger scope than the one in this paper, but the empirical comparison has ignored all of them. \n    - a potential reason for the authors to ignore them is that this new method is considered efficient, but I would recommend the authors to include them in the table, with a comparison of training time together with accuracy. \n\n-  the method is effectively a preprocessing of images, which is instead of preprocessing, is designed for specialized preprocessing with learnable parameters. The kernels are preprocessing of images is well known to the community already, thus, we need more discussions of this method vs. simply preprocessing of the images (although the authors already mentioned data augmentation, but there seems not enough empirical comparisons)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a simple modification to image classification models—a large kernel convolution layer added before existing layers—to significantly improve robustness against various corruptions without needing extra data augmentation, demonstrating notable accuracy increases on ImageNet-100 and ImageNette datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. please offer the comparisions to those methods dedicated to vision robustness, at least for corruptions of images, preferrably with comparisons of computational load, so that there is a better chance this proposed method will stand out. I understand these new methods require huge computational load to compare, but please try to include a comprehensive list of new methods by leveraging reported numbers. This one is the latest one I can find that can give the authors some ideas of the latest method for discussion in their table of comparison [1]. \n\n2. not sure why the ImageNet-1k result is put into appendix instead of the main paper, and also why only Table 2 reports newer methods like ViT and SWIN, but not Table 3, or Figure 4?\n\n3. the result table in appendix is hard to parse, e.g., five groups of results for each corruptions in figures of Section D.6, what is the first group of each corruption? Then, it seems the proposed method does not outperform base methods in the first block for ImageNet-1k, is this CD, clean data? \n\n4. Does this new kernel method involve the tuning of the base model together with the new kernel? There is no guarantee that the kernel can be directly plugged to pretrained weights, right? In this case, the proposed method also have a non-negligible computational load, maybe roughly the same with some regularization based methods, although probably still faster than augmentation-based methods.\n\n[1] Zhou, Andy, et al. \"Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models.\" Advances in Neural Information Processing Systems 36 (2024).", "broader_impact_concerns": "none noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hxvjel5GBQ", "reviewer": "Reviewer_8XnL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Sizable improvements over baseline on standard datasets such as ImageNet-C.\n- Thorough evaluation and ablation of different kernel variants, including interesting insights.\n- Lightweight and applicable post-hoc without data augmentation.\n- Paper nicely written and easy to follow.\n\nWeaknesses and questions:\n- My understanding is that models are all trained from scratch. Did the authors try fine-tuning by just training the initial new layer? I.e., can this be done on pre-trained models. If not, what changes in the other layers to improve robustness?\n- It would be interesting to see evaluations across a wider range of robustness problems such as other ImageNet variants or easy adversarial examples. [1], for example, has a good mix of ImageNet variants to evaluate on. For adversarial examples, simple one-step or black-box attacks would be interesting. Proper adversarial examples will still succeed 100% of the time (is my assumption), but would be interesting if this at least gets a bit more difficult, especially as there are adversarial training results in the appendix (btw, these could also be interesting for the main paper, but might require additional baselines to be interesting and for adversarially trained models, the attacks are too weak, I would expect AutoAttack as evaluation).\n\n[1] https://openaccess.thecvf.com/content/ICCV2023/papers/Guo_Robustifying_Token_Attention_for_Vision_Transformers_ICCV_2023_paper.pdf\n\n- I am also missing some more sensible baseline. I appreciate that the method does not require additional data augmentation. But it would still be valuable to compare to methods like AugMix [2].\n\n[2] https://arxiv.org/abs/1912.02781\n\n- With more baselines, I am curious why the authors did not explore the effect of the proposed layer on top of data augmentation based methods.\n- Experiments with transformer architectures could be in the main paper.\n- The method seems to be worse only on Contrast on ImageNet-C (see appendix), any reasoning behind this?\n\nConclusion:\nI generally like the paper but am a bit hesitant because it is unclear whether the approach scales on top of existing data augmentation strategies or other methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a simple convolutional layer that is prepended to any DNN architecture to improve corruption robustness in image classification tasks. It is shown that having a dimension preserving layer with large kernel size is crucial.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EW2Z3sesm1", "reviewer": "Reviewer_ig1t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The Strengths of the paper are its simplicity, making it easy to be replicated; and the inclusion of ablations using different architectures, including convolutional based, transformers based and mixed models. \n\nThe weakness of the paper can be summarized as:\n- the lack of comparison with other approaches \n- evaluations covering  only tasks with smaller datasets (ImageNet with 9k and Imagenet-100 with 128k) and a small number of classes (10 or 100 respectively). \n- the lack of evidence under stronger training regimes. In the supplementary results, training on the whole ImageNet-1k dataset the performance improvements shown are not as clear, or not presented as the results shown on the main text. Adding to that, results on individual distortions show that the benefit is not shared across severity levels not different distortions.\n\nIn summary, the results shown are valid under the low-data regime, and apparently, ablations cover small models only, but has not shown to be robust under larger/sota training regimes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the use of an extra depth-wise convolutional layer, just after the first convolutional layer. The new component covers a larger neighborhood by the use of a large kernel (25x25) and introduces around 1.9k new parameters. The layers are trained on clean data with no use of augmentation strategies. Results are presented on smaller subsets of ImageNet, namely ImageNette and ImageNet-100.\nThe main contribution claimed is that the addition of such an extra layer increases the robustness across a set of well known classification architectures including: Resnet, Alexnet, Efficientnet, Mobilenet, Densenet, ConvNext, Vit and Swin v2.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- fully describe the architectures ablated. What are the specific version of the models ablated? Which is the VIT,and Swin model adopted? Were the models trained from scratch or finetuned? \n- fully describe the settings for baseline numbers shown. They look particularly low (https://github.com/fastai/imagenette)\n- review the discussion on increase of robustness by the removal of signal content, under stronger training settings (larger dataset and following SOTA best practices).", "broader_impact_concerns": "not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission meets the bar for acceptance on both the Claims and Evidence and Audience criteria. Reviewers also noted its writing quality and clarity (8XnL, ig1t).
Decision: Accept (poster)