As large as it gets – Studying Infinitely Large Convolutions via Neural Implicit Frequency Filters
Authors: Julia Grabinski
Janis Keuper
Margret Keuper
Venue: Accepted by TMLR ()

Abstract: Recent work in neural networks for image classification has seen a strong tendency towards increasing the spatial context during encoding. Whether achieved through large convolution kernels or self-attention, models scale poorly with the increased spatial context, such that the improved model accuracy often comes at significant costs. In this paper, we propose a module for studying the effective filter size of convolutional neural networks (CNNs). To facilitate such a study, several challenges need to be addressed: (i) we need an effective means to train models with large filters (potentially as large as the input data) without increasing the number of learnable parameters, (ii) the employed convolution operation should be a plug-and-play module that can replace conventional convolutions in a CNN and allow for an efficient implementation in current frameworks, (iii) the study of filter sizes has to be decoupled from other aspects such as the network width or the number of learnable parameters, and (iv) the cost of the convolution operation itself has to remain manageable i.e.~we can not naïvely increase the size of the convolution kernel. To address these challenges, we propose to learn the frequency representations of filter weights as neural implicit functions, such that the better scalability of the convolution in the frequency domain can be leveraged. Additionally, due to the implementation of the proposed neural implicit function, even large and expressive spatial filters can be parameterized by only a few learnable weights. Interestingly, our analysis shows that, although the proposed networks could learn very large convolution kernels, the learned filters are well localized and relatively small in practice when transformed from the frequency to the spatial domain. We anticipate that our analysis of individually optimized filter sizes will allow for more efficient, yet effective, models in the future. Our code is available at https://github.com/GeJulia/NIFF .

Published in Transactions on Machine Learning Research (05/2024)
As large as it gets Studying Infinitely Large Convolutions via Neural Implicit Frequency Filters
Julia Grabinski
julia. aoli.ggmoonnsidddmmmmmmmmmm..d
Chair of Machine Learning University of Monnheim
Janis Keuper
kexper@kimla.at Wiimla.
Institute for Machine Learning and Analytics (IMLA) University of Offenbury
Margret Keuper
mwr.et.keuprr@uuimmmmmeee
Chair of Machine Learning University of Mannheim and Mar Planck Institute for Informatics
Saarland Informatics Campus, Germany
Reviewed on OpenReview: /t openrevee.. net/ forum? id= aRy1 jRcHWj
Abstract
Recent work in neural networks for image classification has strong tendency towards increasing the spatial context during encoding. Whether achieved through large convolution kernels or self-attention, models scale poorly with the increased spatial context, such that the improved model accuracy often comes at significant costs. In this paper, we propose a module for studying the effective filter size of convolutional neural networks (CNNs). To fa- cilitate such a sludy, several challenges need to be addressed: (i) we need an effective means to train models with large filters (potentially as large the input data) without increasing the number of learnable parameters, (ii) the employed convolution operation should be plug-and-play module that can replace conventional convolutions in CNN and allow for an efficient implementation in current frameworks, (iii) the study of filter sizes has to be decoupled from other aspects such as the network width or the number of learnable pa- rameters, and (iv) the cost of the convolution operation itself has to remain manageable ie. we can not naïvely increase the size of the convolution kernel. To address these chal- lenges, propose to learn the frequency representations of filter weights as neural implicit functions, such that the better scalability of the convolution in the frequency domain can be leveraged. Additionally, due to the implementation of the proposed neural implicit func- tion, even large and expressive spatial filters can be parameterized by only a few learnable weights. Interestingly, our analysis shows that, although the proposed networks could learn very large convolution kernels, the learned filters are well localized and relatively small in practice when transformed from the frequency the spatial domain. We anticipate that our analysis of individually optimized filter sizes will allow for more efficient, yet effective, models in the future.
1 Introduction
Recent progress in image classification such as Liu et al. (2022b) builds upon observations on the behavior of vision transformers Dosovitskiy et al. (2020); Khan al. (2022); Touvron al.) (2021): ; Vaswani et al.) (2017), which rely on the learned self-attention between large image patches and therefore allow information
from large spatial contexts to be encoded. Tn particular, have recently seen trend towards increasing

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
Vi
V2
V3
V4
V5
V6
56x56
NIF
0.75
Layer 1 zoomed
9x3
0.62
0.17
0.10
0.04
0.04
0.01
0.01
Standard 3x3
0.25
0.25
0.14
0.14
0.12
0.10
0.08
0.06
NUFT 14x14
-0.25
Layer
0.87
0.09
0.02
<0.01
<0.01
<0.01
<0.01
Standart
3x3
-0.75
0.69
0.14
0.07
0.03
0.03
0.01
0.01
Figure 1: PCA components of learned kernel weights in the first layer of a ResNet50 trained ImageNet- - the 1st row shows the learned NIFF kernels transformed to the spatial domain. Row visualizes a zoomed-in version with explained variance for each component. The 3rd row shows PCA and explained variance for a standard CNN with the standard kernel size 3 x 3. The 4th row shows that NIFF actually learns large and highly structured spatial filters for the 4th layer of the same network, while the baseline model is limited small filters. This PCA analysis is one of the tools apply to answer the central question of this paper: How large do CNN kernels really need to be? It demonstrates that networks which can learn filter kernels large their featuremaps, still learn well-localized, small kernels. Yet, these kernels larger than the typically applicd 3 x 3 kernels.
the spatial context during encoding in convolutional neural networks (CNNs), leading to improved results for CNNs accordingly, as for example in Guo et al. (2022;: Liu et al. (2022ah; Ding al. (2022): Peng al. (2017) Yet, model parameters and training times scale poorly with the filter size, such that the increased model accuracy often comes at significant costs if no additional model-specific tricks are applied. At the same time, it remains unclear whether there is an optimal filter size and which size of filters would be learned, could the models learn arbitrary sizes.
This paper aims to provide such a study and the corresponding tool that can modularly replace the convo- lution operation in any CNN architecture allowing for the efficient training of arbitrarily large convolution kernels be analyzed. However, efficiently training models on standard datasets such as ImageNet Deng et al. 2009) with large filters (potentially large the input data) is non-trivial. Not only should the number of parameters remain in range comparable to the one of the baseline models, but also has the cost of the actual convolution operation to remain manageable. Neural implicit functions, such as previously used in Romero et al. (202aa); Sitzmann et al. )2000), can limit the number of learnable parameters while learning large convolution filters. Yet, their evaluation limited to low-resolution data because of the poor scalability of the convolution operation itself, ie. increasing the size of the learned convolution filters directly implicitly not scalable solution. Therefore, we propose to learn filters in the frequency domain via neural implicit functions. This has several advantages: First, the convolution operation can be executed in the Fourier domain, where it scales significantly better with the filter size. Second, due to the implementa- tion of the neural implicit function, the number of lcarnable model parameters remains similar to that of the baseline model. Third, the learned filters are directly expressed in the Fourier domain i.e. as oriented sine and cosine waves. Thus, highly structured periodic spatial convolution kernels can be learned using small MLPs with only a few parameters. Our proposed Neural Implicit Frequency Filter (NIFF) convolution operation is a plug-and-plas -lay module that can replace any conventional convolution in a CNN and allows for an efficient implementation in current frameworks. The resulting neural implicit frequency CNNs the first models to
2

---

Published in Transactions on Machine Learning Research (05/2024)
achieve results on par with the state-of-the-art on standard high-resolution image classification benchmarks while executing convolutions solely in the frequency domain. Thus, NTFFs allow us to provide an extensive analysis of the practically learned filter sizes while decoupling the filter size from other aspects such as the network width or the number of learnable parameters. Interestingly, our analysis shows that, although the proposed networks could learn very large convolution kernels, the learned filters practically correspond to well-localized and relatively small convolution kernels when transformed from the frequency to the spatial domain. Our contributions can be summarized as follows:
We present novel approach which enables CNNs to efficiently learn infinitely large convolutional filters. To do so, we introduce MLP parameterized Neural Implicit Frequency Filters (NIFFs) which learn filter representations directly in the frequency domain and can be plugged into any CNN architecture.
Empirically, we show that NIFF's facilitate model performance on par with the baseline without any hyperparameter tuning. Hence, our proposed frequency filters allow, for the first time, to efficiently analyze filter sizes and encoded context sizes via filters that have potentially an infinite extent in the spatial domain.
Finally, we analyze the spatial representations of the resulting large filters learned by various CNN architectures and show very interesting results in terms of practically employed spatial extent.
2 Related Work
Large kernel sizes. In recent years, vision transformers have facilitated significant. improvement in image classification Dosovitskiy et al. (2020); Vaswani et al. (2017) and related tasks. Such models are based on larger image patches and self-attention, i.e. they allow for the entire spatial context to be encoded. Subsequently, Liu et al. showed that smaller receptive fields by shifted window attention with window sizes of 7 7 to 12 x 12 can improve network performance, while follow-up works again increased the window sizes Dong et al. 2022). What remains is the general observation that image-processing models can benefit from encoding larger spatial context, especially in deeper network layers. This holds also for pure convolutional models as shown in Liu et al. (2022b) with 7 sized depth-wise convolutions that can even outperform transformer models. Guuo et al. and Peng et al. further increase the receptive fields of the convolution and Ding et al.] and Liu et al. achieve improved results with 31 31 and even 61 x 61 sized convolution filters, respectively. To facilitate scaling, Ding et al. use depth-wise convolutions instead of full convolutions and thereby increase the model size by only 10-16%. Liu et al.) decompose the 61 x 61 convolution into two parallel and rectangular convolutions to reduce the computation load and parameters. Further, Agnihotri et al. (2023) showed that increased spatial context can improve the upsampling decoder networks. Similarly, Jung & Keuper (2021): ; Durall al. (2020) counteract spectral artifacts during image generation using an adversarial loss and regularization combined with large filter sizes, respectively. To allow to further scale even models using depth-wise convolutions, NIFFs directly learn and execute the convolution in the frequency domain. An additional complication of large spatial convolutions are potential spectral artifacts that can be handled e.g. by Hamming windows Tomen & van Gemert (0011).
In sequence modelling, so-called long convolutions also enable models to encode larger parts of a sequence at low cost. Several works Poli al. (2023); Romero et al. (202b)); Gu et al. (2022) make use of the convolution theorem to model global large 1D convolutions. [ao al. propose Global Filter Networks based on Swin- Transformers. They encode intermediate layers as convolutions in the frequency domain followed by a feed-forward neural network, and thus facilitate efficient convolutions with infinite extent. They achieve very good results on ImageNet-lk. However, they face the computational overhead of lcarning the filter weights directly, resulting in an increase in the number of parameters. We also compute convolutions in the frequency domain to benefit from the better scalability of the convolution operation. Yet, our NIFF module uses neural implicit functions to preserve the original number of model parameters. The model proposed by Rao et al. is over six times larger than ours. Furthermore, our study focuses on the analysis of the effective kernel size for convolutions employed in CNN backbones and not on Transformer architectures.
3

---

Published in Transactions on Machine Learning Research (05/2024)
Dynamic and steerable filters. While our approach focuses on the evaluation of the chosen filter size of the network given that it can learn infinitely large filters in theory, dynamic filtering tries to directly address this challenge by e.g. deformable convolutions Dai et al. (2017). Unlike traditional convolutions where the kernel is fixed and applied uniformly across the input, deformable convolutions allow the filter to adapt its shape and size based on the input data. This is done via a learnable offset, that controls the sampling locations of the convolutional operation within the input feature map Further, Pintea et al. (2021) proposod to use flexible kernel sizes at each stage by learning the values for a combination of Gaussian derivative filters thus inherently controlling the size of the filter. [Sosnovik et all (2019) combine the concepts of steerable filters Cohen & Welling (2016) and group equivariance to achieve scale-equivariant representations, allowing it to effectively handle objects at different scales without the need for explicit scale normalization techniques. Similarly [Worrall & Welling) (2019) aims to achieve equivariance over scale. They inlegrate scale-space operators, such as Gaussian blurring and Laplacian of Gaussian operators, into the network layer to be more robust to variations in object size or resolution images.
Training CNNs in the Frequency Domain. Most works implementing convolutions in the frequency domain focus on the efficiency of the time and memory consumplion of this operation Ayat et al. (2019); Guan et al. (2019); Mathieu et al. (2013); Pratt et al. (2017); Vasilache et al. (2014); Wang et al. (2016). since the equivalent of convolution in the spatial domain However, most of these approaches still learn the convolution filters in the conventional setting in the spatial domain and transform featuremaps and kernels into the frequency domain to make use of the faster point-wist multiplication at inference time Ayat et al.] (2019): Mathieu et al. (2013); Wang et al. (2016). This is mostly due to the fact that one would practically need to learn filters as large as featuremaps when directly learning the frequency representation. Those few but notable works that also propose to learn convolutional filters in CNNs purely in the frequency domain, until now, could only achieve desirable evaluation performances on MNIST and reach low accuracies on more complex datasets like CIFAR or ImageNet Pan et al. (2022): ; Pratt et al.] (2017): Watanabe & Wolf (2021) if they could afford to evaluate on such benchmarks at all. Tn Table 1 we give brief overview on some of these approaches, executing the convolution or even more components in the frequency domain. As shown in Table A9 in the appendix, NIFF can also be combined with other modules in the frequency domain.
point-wisc multiplication in the froquency domain.
Further approaches apply model compression by zeroing out small coefficients of the featuremap in the froquency domain Guan et al. (2019), enhance robustness by frequency regularization Lukasik et al. (2023) or enrich conventional convolutions with the frequency perspective to get more global information Chi el al.
(2020).
In contrast to all these approaches, neither aim to be more time or memory-effiiiet nor do we want to boost performance via additional information. Our aim is to investigate which filter size CNNs practically make use of if they have the opportunity to learn infinitely large kernels. To do so, we propose a model that facilitates learning large filters directly in the frequency domain via neural implicit functions. Thus, we are the first to propose a CNN whose convolutions can be fully learned in the frequency domain while maintaining state-of-the-art performance on common image classification benchmarks.
Neural Implicit Representations. Neural implicit representations generally map a point encoded by coordinates via an MLP to a continuous output domain. Thus, an object is represented by a function rather than fixed discrete values, which is the more common case, e.g., discrete grids of pixels for images, discrete samples of amplitudes for audio signals, and voxels, meshes, or point clouds for 3D shapes Chibane et al. (2020 Sitzmann et al. (2020). Continuous representation of the signal, i.e. neural implicit representations, provide fixed number of parameters and are independent of spatial frequency resolutions. Moreover, neural implicit representation can also be used to learn to generate high-resolution data Chen et al. (2021) to learn 3D representations from 2D data [a et al. (2022) and Ma et al. (2023) introduce hyperconvolutions for biomedical image segmentation Based on the concept of Hypernetworks Ha el. al. (2016), they learn a predefined SIREN Sitzmann et al.) (2020) consisting of stacked Ix1 convolutions and periodic activations to predict the spatial kernel weights with prodefined size. Similarly, Romero et al. (202aa) introduces continuous kernels by learning spatial kernel weights that have the same size as featuremaps with a Hypernetwork consisting of stacked 1x1 convolutions and then learn to mask and crop the actual filter size. All their
4

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Overview of different approaches operating in the frequency domain compared to our NIFF. The approaches in the first two rows (Mathieu et al. (2013); Wang et al. (2016)) focus the time improvement of the point-wise multiplication compared to standard convolution. The third and fourth proaches that operate (almost) fully in the frequency domain (Pratt et al. (017) Pan et al. (2022) ). Tomen & van Gemert (2021) applies a global filter layer via point-wise multiplication in the frequency domain at the stem of a transformer model. In comparison we present NIFF, a plug-and-play operation that can replace any kind of convolution operation learn and infer fully the frequency domain. Note that these met have been proposed and optimized for different purposes and the respective numbers on CIFAR-10, where reported, are not comparable.
present ap-
Method
Architecture not reported not reported
Operations the frequency domain reported CIFAR-10 acc
Fast FFT 2013
Executing of the Convolution
no acc. reported acc. reported
CS-unil 2016
Couvolution + Downsampling
FCNN
FCNN CEMNet
Full Network
23% 59.33%-78.337
CEMNet
et
2022
Full Network except MaxPooling
GFN
202
Swin- Transformer
First Patchifying operation
98.60%
NIFF (ours)
any CNN
Convolution
90.63%-94.0039
Spatial Large Convolution
Neural Implicit Filter naive implementation
Neural Implicit Frequency Filter (NIFF) (ours)
for in N for we in N
11g
stackked TK1 Convolutions
G. mulisilation weights size x N
MLP
C₂ mulipi laaton
RC,
weights
Iteratively querying the MLP N times
once executed
x C2
fixed numbers parameters
fixed numbers of parameters
numbers of parameters
Figure 2: While learning large kernels increases the number of Icarnable paramcters quadratically (here NxN (left)), neural implicit functions use a fixed amount of learnable parameters (middle). When using a simple MLP, the input a 2D vector containing the wi and wy coordinates of the desired filter position. Our NIFF (right) implements the MLP efficiently using several 1x1 convolutions which start with an input channel size of two, encoding the wa and wy direction. Hence, there no need to iterate over each coordinate separately. Following, include some hidden layers and activations to learn and at the last layer, the number of output channels is set to the desired number of clement-wise multiplication weights.
operations are applied in the spatial domain. To facilitate the learning of suitable filter structures, they replace the periodic activation of the SIREN by Multiplicative Anisotropic Gabor Networks. In contrast, our NIFFs directly learn the Fourier representation of convolution filters, i.c. representation in a basis of sine and cosine waves, using an MLP with conventional activations. Thus, we can efficiently execute the convolution in the Fourier domain with the objective of investigating the effective spatial extent of the learned convolution filters.
3 Convolutions in the Frequency domain
In this paper, we leverage neural implicit functions in a novel setting: we learn neural implicit representations of the spatial frequencics of large convolution kernels. This allows us employ the convolution theorem from signal processing Forsyth & Ponce (2003) and conduct convolutions with large kernels efficiently in the frequency domain via point-wise multiplications.
5

---

Published in Transactions on Machine Learning Research (05/2024)
Small Spatial depth- wise Convolution
Large Spatial depth- wise Convolution
Depth-wise NIFF (ours)
NkN featuremaps
*
NxN featuremaps
*
NkN kernels
NxN featuremaps
NxN multiplication weights
3x3 kemels
N(3
O
((N))
Figure 3: Concept of our NIFF Convolutions and the complexity for each operation, explained with the example of a depth-wise convolution. (Left) The standard depth-wise convolution in which we have as many kernels as we have featuremaps. Each kernel is convolved with a featuremap. (Middle) Large convolution with kernels as large as the featuremaps. (Right) The NIFF convolution which simply applics a pointwise multiplication between the FFT-transformed Teaturemaps and the learned kernel weight via our NIFF. The newly updated featuremaps are transformed back into the spatial domain via inverse FFT (IFFT).
Properties of Convolutions in the Frequency Domain. According to the convolution theorem Forsyth & Ponce 2000),, circular convolution, denoted by @, between signal g and filter & in the spatial domain can be equivalently represented by point-wise multiplication, denoted by @ of these two signals in the frequency domain, for example by computing their Fast Fourier Transform (FFT), denoted by the function F() and then, after point-wise multiplication, their inverse FFT F-)
g 2 (F(=)
(1)
While this equivalence has boen less relevant for the relatively small convolution kernels employed in tra- ditional CNNs (lypically 3 or at most 7 7); it becomes highly relevant as the filter size increases to maximum: The convolution operation in the spatial domain is in N(M)NN) for discrete 2D signal g with N x N samples and filters & of size M x M, i.c. O(N)) when discrete filters k have the same size as the signal g. In contrast, the computation is in O(N"log(N)) when executed using Fast Fourier Transform (FFT) Cooley & Tukey 19655) and point-wise multiplication according to Equuaion equation 1 for the FFT according to Cooley & Tukey (965)) is given in the appendix.
The proof
Thus, for efficient filter learning, we assume that our input signal, ie. our input image or featuremap, is given in the frequency domain. There, we can directly learn the element-wise multiplication weights correspond- ing to F(k) in Equation equation | and thereby predict infinitely large spatial kernels. These element-wise multiplication weights m act like a circular convolution which can be seen as an infinite convolution due to the periodic nature of the frequency domain representation. This means in practice that if we represent a signal, in our case an image or a fcaturemap, in the frequency domain and transform it back into the spatial domain, we assume that this signal is periodic and thus infinitely large. For many images, such boundary conditions can make sense since the image horizon line is usually horizontally aligned.
Practically, the kernels applied in the frequency domain are band-limited to the highest spatial frequency that can be represented in the featuremap. However, since higher frequencies can not be encoded in the discrete input signal by definition, this is no practical limitation. In the frequency domain, we can thus efficiently apply convolutions with filters with standard sizes of for example 224 x 244 (for ImageNet) or 32 x 32 (for CIFAR-10) learned spatial frequencics, solely limited by the resolution of the input featuremap.
3.1 Neural Implicit Frequency Filters
Images are lypically stored as pixel-wise discrete values. Similarly, the filter weights of CNNs are usually learned and stored as discrete values, i.e. a 3 x 3 filter has 9 parameters to be learned and stored, a 7 x 7 filter as in ConvNeXt has 49 parameters and filter as large as the featuremap would require e.g. 224 224 (50176) parameters for ImageNet-sized network input. In this case, it is not affordable to directly learn these filter weights, neither in the spatial nor in the frequency domain. To address this issue, we propose to parameterize
6

---

Published in Transactions on Machine Learning Research (05/2024)
filters by neural implicit functions instead of learning the kernels directly. This is particularly beneficial since we can directly learn the neural implicit filter representations in the frequency domain. Figure 2 depicts the benefit of neural implicit functions for large kernel sizes.
Thus, formally, we learn a function, F parameterized by that takes as input the spatial frequency (Wg. wy) whose filter value will predict,
F6 Fa:R² + CC, .m(_,, wy) = Fo(wa,wy)
(2)
where C' is the number of filter channels and the complex-valued m(w,,wy) in dimension e is the c-th filter value in the frequency domain to be multiplied with F(g)(W,, for featuremap 9- Specifically, with the implicit function Fo, we paramcterize the weights with which the featuremaps are multiplied in the frequency domain based on equation by point-wise multiplication. The number of MLP output channels is equivalent to the number of channels C' for the convolution, and its hidden dimensions determine the expressivity of each learned filter, which we term Newral Implicit Frequency Filter (NTFF). In practice, the complex and real valued parts of NIFFs are independently parameterized.
While neural implicit functions allow parameterizing large filters with only a few MLP model weights, their direct implementation would be highly inellicient. Therelore, we resumee to trick that allows efficient training and inference using standard neural network building blocks. Specifically, we arrange the input to the MLP, i.. the discrete spatial frequencies (Wz.wy) for which we need to retrieve filter values, in 2D arrays that formally resemble featuremaps in CNNs but are fixed for all layers. Thus, the MLP takes one input matrix encoding the æ coordinates and one encoding the y coordinates as shown in Figure 3 Then, the MLP can be equivalently and efficiently computed using stacked 1 convolutions, where the first x 1 convolution has input depth two for the two coordinates, and the output layer 1 x 1 convolution has C' outpul dimensions.
Efficient Parameterization of NIFFs.
3.2 Common CNN Building Blocks using NIFF
Well-established models like ResNet He et al. 2016) use full convolutions, while more recent architectures employ depth-wise and 1 x 1 convolutions separately Liu et al. (2022b). Our neural implicit frequency filters can be implemented for all these cases. However, operations that include downsampling by stride of two are kept original spatial convolution. In the following, we describe how commonly used convolution types are implemented using NIFF.
The NIFF module for the depth-wise convolution is as follows: First, we transform the featuremaps into the frequency domain via FFT. Afterwards, the learned filters are applied via element-wise multiplications with the featuremaps. Thereafter, the featuremaps are transformed back into the spatial domain via inverse FFT (if the next step to be executed in the spatial domain). The entire process is visualized in Figure 3 As discussed above, this process allows to train models with large cércular convolutions in a scalable way. An ablation circular versus linear convolutions in the frequency domain is given in the appendix in Table A3
Depth-wise Convolution.
Full Convolution. To employ NIFF for full convolution 2DConv with Cp imput channels and Cg output channels the convolved featuremaps with the kernel weights need to be summed according to
Cp 2DConv(ge,, kc, = ko = 9C,
(3)
Conveniently, summation in the spatial domain is equivalent to summation in the frequency domain and can be performed right away.
g F F(k))
(4)
The full convolution in the frequency domain can be implemented by first predicting the frequency represen- tation of kc,.c, directly using NIFF, i.e. for each output channel. Then, all input channels are element-win

---

Published in Transactions on Machine Learning Research (05/2024)
multiplied with the filter weights and summed up in 2DConv
C2 Σ Dc @0 kic,o =
C2 Fled) o Flhacdd
(5)
Yas the NIFF needs to output Cp x Cg instead of only Cg at its last layer leading to significant increase of learnable parameters for our NIFF. Thus, for efficiency reasons, we docompose the full convolution into depth-wise convolution followed by a 1 x 1 convolution in practice. The transformation into the frequency domain is applied before the depth-wise convolution, where the backward transformation into the spatial domain is applied after the 1 x convolution. While not equivalent, the amount of learnable parameters decreases significantly with this approach, and the resulting models show similar or better performance in practice. An ablation on full convolutions versus depth-wise separable convolutions is provided in Table 2 and in the appendix in Table Al
1 1x1 1 Convolution. To perform 1 x convolution, we transform the input into the frequency domain via FFT. Afterwards, we apply linear layer with channel input neurons and desired output dimension output neurons on the channel dimension. Finally, we transform back into the spatial domain via inverse FFT. While spatial 1 x convolutions only combine spatial information in one location, our x in the frequency space is able to combine and learn important information globally.
Other operations such as downsampling, normalization, and non-linear activation are applied in the spatial domain so that the resulting models are as close as possible to their baselines while enabling infinite-sized convolutions.
4 NIFF Model Evaluation
We evaluate variety of image classification CNNs with our proposed NIFF. Overall, we achieve accuracies par with the respective baselines. For high-resolution data, our NIFF CNNs perform slightly better than the baseline models, while the large kernels are apparently less beneficial for low-resolution data, especially in deeper layers where the featuremap sizes are very small. Further, we evaluate the performance of NIFF when combined with other methods applied in the frequency domain in the appendix (Appendix E
Results on Image Classification. For high-resolution datasets like ImageNet-Ik 1k or ImageNet-1100, NIFF CNNs are able achieve results on par with the baseline models. Table 2 and Table A1 in the appendix report these results. For models which originally employ full 2D convolutions, we also ablate on the effect of replacing these with desth-wise separated convolutions in spatial domain with the original filter size and for the smaller ResNet-18 and ResNet-50 models, we report results on NIFF for full 2D convolution. In line with the finding e.g. in Liu et al. (2022b), the differences are rather small while the amount of parameters decreases significantly as depth-wisc separable convolutions are applied. According to our results, this is equally true for both, convolutions executed in the spatial and in the frequency domain. For comparability, we simply used the baseline hyperparameters reported for each of these models in the respective papers, i.e. we achieve results on par with the respective basclincs without hyperparameter optimization. For different ResNet architectures, even achieve improvements both high-resolution datasets.
The complete results on CIFAR-10 are reported in Table A2 in the Appendix. Here, we observe a slight drop in performance, which may be due to the low spatial resolution of images and featuremaps. Particularly in deeper layers, the potential benefit from large convolutions is therefore limited. Further, we want to emphasize that NIFF not conceived to improve over baseline methods (we are of course glad to observe this tendency for high-resolution data) but to facilitate the evaluation of effective kernel sizes in CNNs.
5 How large do spatial kernels really need to be?
After empirically validating our proposed NIFFs, we now quantitatively analyze how large the spatial kernels really tend to be. Hence, we transform the learned filters into the spatial domain and plot the relative density
8

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: Evaluation of top 1 and 5 test accuracy on ImageNet- 1k and ImageNet-100 for different network architectures. We used the standard training parameter for each architecture and the advanced data augmen- lation from Liu et al. (2022b) for all architectures. Additionally, for the timm baseline models on ImageNet- 1k for each network. More ablations on ImageNet- 100 regarding more architectures and evaluating linear convolutions are given in the appendix Table A1 and Table A3
add the numbers reported by Pytorch
ImageNet-lk
ImageNet-100
Model
# Params
Acc@1
Acc@5
# Params Acc@1
Acc@5
ResNet-18 He et al.
2016 (Pytorch)
11.689.512
69.76
89.08
11.227.812
ResNet-18 He et. al.
2016
11.689.512
72.38
90.70
11.227.812
87.52
97.50
ResNet-18 NIFF 2D
cOnv (ours)
21.127.320
71.00
89.95
20.665.620
86.42
97.08
ResNet-18 separated conv
3.327.400
69.12 12
88.84
2.865.700 700
86.52
97.20
ResNet-118 NIFF (ours)
3.660.360
70.75
90.01
3.198.660
86.52
97.14
ResNet-50 He et al.
2016
(Pytorch)
33.557.032
76.13
92.86
23. 712.932
ResNet-50 Hc ct al.
2016
25.557.032
79.13
94.43
23.712.932
89.88
98.22
ResNet-50 NIFF 2D
conv (ours)
33.778.328
78.65
94.25
31.934.228
90.08
98.06
ResNet-50 separated conv
18.275.688
77.76
93.72
16.431.55
89.78
98.18
ResNet-50 NIFF (ours)
18.605.000
79.65
94.80
16. 760.900
89.98
98.44
ResNet-101 He et al.
2016) (Pytorch)
44.549.160
77.37
93.55
42705.060
ResNet-101
He et al.
2016
14.549.160
80.63
95.11
42. 705.060
90.54
98.14
RosNet-101 scparated conv
28.394.088
79.53
94.64
26.549.988
90.20
98.36
ResNet-101 NIFF (ours)
29.187.431
80.26
95.23
27.343.332
90.54
98.38
DenseNet-121 Huang et al.
2017
(Pytorch)
7.978.856
76.65
92.17
7.978.856
DenseNet-121 Huang et al.
2017
7.978.856
75.11
92.50
7.056.356
90.06
98.20
DenseNet-121 NIFF 2D conv (ours)
10.119.752
73.96
91.82
9.197.252
89.66
98.32
DenseNet-121 separated conv
6.145.128
69.80
89.02
5.222.628
89.94
98.08
DenseNet-121 NIFF (ours)
6.159.512
74.58
92.33
5.237.012
90.24
98.18
Conv-eett-iny
Liu et al.
(002bb (022b
(PyTorch)
28.5 28.589.128
82.52
96.15
28.589.
ConvNeXt--tnny
Liu et al.
28.589.128
(1.897.028
91.70
98.32
ConvNeXt-tiny NIFF (ours)
28.890.66)
81.83
95.80
92.00
98.42
MobilcNet-v2
Sandler et al.
2018
(Pytorch)
3.504.872
71.88
90.29
3.504.872
MobileNet-v2
Sandler et al.
2018
3.504.872
70.03
89.54
2.351.972
84.06
96.52
MobileNet-v2 NIFF (ours)
3.512.560
71.53
90.50
2.359.660
85.46
96.70
of each spatial kernel k, i.c. the ratio of the kernel mass that contained within centered, smaller sized, squared kernels. The kernel has the same width and height as the featuremap (FM) it is convolved to.
width height
k ( width
height
+
+ww W.
+
2
2
kernel mass ratio ratio(width, (width,
height) height)
(6)
PMwidth FMheight
Σ
(k(w,h)
where is the center of the full kernel & in the spatial domain. In Figure 4 we report the kernel mass ratio for different networks trained on 1nk. We observe that all networks mostly contain well-localized kernels that are significantly smaller than possible. Yet, the first. layer in DenseNei-121 and the second and third layer in ResNet-50 also contain larger kernels that make use of the possible kernel size up to 56 x 56 and 28 28 respectively. For MobileNet-v2, the spatial kernels are prodominantly well-localized and small. However, the maximal resolution of 112 112 some kernels are quite large, at least 56 56 ( (5%) indicating that MobileNet-v2 which uses small 3 x kernels could benefit from larger kernels. Similar results

---

Published in Transactions on Machine Learning Research (05/2024)
ResNet-50
DenseNet-121
1.0
kernel mass ratio
kernel mass ratio
Layer (max resolution 56x56)
Layer (max reslution 56x56)
0.4
Layer 2 (max resolution 56x56)
Layer (max resolution 28x28)
Layer 3 (max resolution 28x28)
Layer (max resolution 14x14)
0.2
Layer 4 (max resolution 14x14)
0.2
Layer (max resolution 7x7)
effective kernel size 9x9
effective kernel size 9x9
0.0
0.0
10
20
30
40
50
10
20
30
40
50
kernel size
kemel size
MobileNet-v2
ConvNeXt-tiny
1.0
1.0
kernel mass ratio
0.8
kernel mass ratio
0.8
0.6
max resolution 112x112
0.6
max resolution 56x56
Layer (max resolution 56x56)
0.4
resolution 28x28
0.4
Layer (max resolution 56x56)
max resolution 14x14
Layer (max resolution 2 28x28)
0.2
max resolution 7x7
0.2
Layer (max resolution 14x14)
effective kemel 9x9
effective kernel size
0.0
0.0
20
40
60
80
100
30
40
50
kernel size
kemel size
Figure 4: Effective kernel size evaluation ImageNet-Ik. We plot the average ratio of the entire kernel mass contained within the limited spatial kernel size, where the x-axis denotes the width and height of the squared kernels. For ResNet-18, ResNet-50 and ConvNeXt-tiny each layer encodes one resolution. Thus, these worko' layers could be summarised (Tayer encoding 56 x 56, Layer 28 x 28, Layer 3 14 x 14 and Layer 7 x 7). However, for MobileNet-v2 the resolution downsampled within a layer.
on ImageNet-100 are reported in the Appendix (Figure 2022a), who investigate the spatial kernel size of CNNs in the spatial domain.
A2 These findings are in line with Romero et al.
In Figure 5 we evaluate the spatial kernel mass ratio of different networks trained on CIFAR-10. For all models, we see a clear trend. They learn in all layers well-localized, small kernels. Similar to the ResNet trained on ImageNet-Il, the learned kernels barely exceed the size of 5 x 5. In contrast, some of the learned spatial kernels by MobilcNet-v2 and ConvNeXt-tiny use the full potential for cach kernel size, respectively.
6 Filter Analysis
In this section, we visualize the spatial kernels learned by NIFF. We do so by transforming the learned multiplication weights via inverse FFT into the spatial domain. Afterwards, apply Principle Component Analysis (PCA) per layer to evaluate the predominant structure of the learned spatial kernels. For each layer, we separately plot the six most important eigenvectors and zoom in to visualize the 9 x 9 center (see Appendix Fig. A8 for the full visualization without center cropping). The original spatial kernels are shown in the Appendix (Fig. A22 and A23
Our results indicate that the networks, although they could learn infinitely large kernels, learn well-localized, quite small (3 x 3 up to 9x 9) kernels especially in early layers. The spatial kernel size is different for low- and high-resolution data. Figure 6 visualizes the eigenvectors of the spatial kernel weights for each layer in our NIFF CNN for ResNet-50 trained on ImageNet-1k. All layers dominantly learn filters with well-localized, small kernel sizes. Especially in the later layers, most variance is explained by simple 1 x kernels, whereas the first-layer kernels have the most variance explained by simple x or 7 x 7 kernels. Hence, the networks learn larger kernels than the standard x 3 kernels used in a ResNet. However, the full potential of possible sizes of 56 x 56 and more is not used. Similar results are shown in Figure D as well as for more architectures (Figure A5 A6 and A7 and ImageNet-100 (Figure Ag
A10
A11 and A12 in the Appendix with consistent
10

---

Published in Transactions on Machine Learning Research (05/2024)
ResNet-18 18
ResNet-50
kernel mass ratio
kernel mass ratio
Layer 1 (max resolution 16x16)
(max resolution 16x16)
Layer 2 (max resolution 8x8)
(max resolution 8x8)
0.2
Layer 3 (max resolution 4x4)
0.2
Layer (max resolution 4x4)
effective kernel size 5x5
effective kernel size
5x5
0.0
2
à kernel size
12
14
16
0
4
10 kemel size
12
14
16
MobileNet-v2
ConvNeXt-tiny
1.0
kernel mass ratio
0.8
kernel mass ratio
0.8
0.6
0.6
max resolution 32x32
Layer (max resolution 32x32)
0.4
resolution 16x16
0.4
Layer (max resolution 16x16)
max resolution effective
Layer (max resolution 8x8)
0.2
4x4
0.2
Layer resolution 4x4)
effecive
0.0
0.0
10
15
20
25
30
5
10
15
20
25
30
kernel size
kemel size
Figure 5: Effective kernel size evaluation on CIFAR-10. We plot the average ratio of the entire kernel mass contained within the limited spatial kernel size, where the x-axis denotes the width and height of the squared kernels. For ResNet models, cach layer encodes one resolution. Thus, the layers could be summarised (Layer encoding 16 16, Layer 2 8 8 and Layer 4 x 4). For ConvNeXt-tiny the first layer started with 32 32, However, for MobileNet-v2 the resolution is downsampled within layer.
Vo
V1
V2
V3
V4
V55
V6
1.00
Layer
56x56
0.75
0.62
0.10
0.04
0.04
0.01
Layer 2
56x56
0.25
0.75
0.15
0.04
0.01
0.01
0.01
<0.01
0.00
Layer 3 28x28
0.25
0.90
0.06
0.02
<0.01
<0.01
<0.01
-0.50
14x14
Layer
0.87
0.09
0.02
<0.01
<0.01
<0.01
<0.01
-1.00
Figure 6: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of ResNet-50 trained on ImageNet-lk zoomed to 9. On the left, the maximal filter size for the corresponding layer is given. We can see that most filters only use a well-localized, small kernel size although they could use a much bigger kernel.
results; all networks learn well-localized, small kernels, even though they could learn much larger ones. Some exceptions can be observed for the 4th layer, where the eigenvector with the second highest explained variance still larger extent of the kernel. In combination with the downsampling applied prior to the
11

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V2
V4
V6
1.00
16x16
Layer
+
0.75
0.85
0.13
0.01
0.01
<0.01
<0.01
<0.01
0.50
NIFF Layer 8x8
0.78
0.07
0.06
0.05
0.03
0.01
0.01
0.00
Layer 4x4
-0.25
0.72
0.10
0.09
0.07
0.03
<0.01
<0.01
-0.50
Standard Layer 3 3x3
0.25
0.18
0.13
0.12
0.09
0.08
0.06
Figure PCA basis and explained variance for each basis vector (below) of all spatial filters learned by NIFF for each layer well the learned filters for the third layer of standard ResNet-18 trained CIFAR-10. On the left, the layer and its filter size are given. Most filters only well-localized, small kernel size although they could use much bigger kernel.
10a2
Convolution
FLOPs
1030
FFT IFFT
10a
linear Convalution
100
200
300
400
500
feature mapifllter size
Figure 8: FLOPs in Log-scale for computing a simple FFT and IFFT, a standard depth-wise convolution and our NIFF (including FFT and IFFT) and a lincar convolution executed with out NIFF for convolutions with kernels as big as the featuremape for the example of 64 channels.
fourth layer, this indicates that the network tries to keep up large spatial context. The effect be observed for networks trained on CIFAR-10. Figure 7 shows the PCAs of the spatial kernel weights for each layer in NIFF CNN for ResNet18 trained on CIFAR-10. There the dominant basis vectors for the first layer learned by our NIFF kernels do not exceed a size of 3 x 3 in the spatial domain. However, in the second layer, a few kernels (19%) are larger than 3 3 mostly x 7. In the last layer, the kernels again do not exceed the standard size of 3. Similar results on further networks are presented in the Appendix (Figure A13)
Compute Costs. Since our NIFF implementation conceived for analysis purposes, our models are not optimized for runtime. In particular, compute repeated FFTs in Pytorch to allow the computation of the remaining network components in the spatial domain, so that the models are equivalent to large kernel models computed in the spatial domain. Yet, Figure S demonstrates that with large kernel sizes our NIFF approach with repeated FFIs is much more efficient in terms of FLOPs compared to the spatial convolution. Further runtime evaluations are reported in the Appendix (Tables A5 and A6
7 Discussion and Conclusion
With the help of the proposed NIFF, we can analyze the effectively learned kernel size of state-of- soatte--o-thh-art CNNs the spatial domain. We could observe that the full potential for much larger receptive fields is not used in most models. Especially for low-resolution datasets like CIFAR-10, the network refuses learn much larger kernels than 3 3. However, in high-resolution datasets such ImageNet, the models indeed use
12

---

Published in Transactions on Machine Learning Research (05/2024)
larger kernel sizes than x but still the majority of filters is not much larger than 9x9. Therefore, networks with small kernel sizes such as ResNet He et al. (2016) can benefit from convolution kernels through NIFFs, while networks such as ConvNeXt Liu et al. (2022b) are already close to optimal for ImageNet in terms of used kernel size. However, we also find that there is no globally optimal filter size, as the networks also learn a small amount of large kernels. Thus, we might need to rethink the current state-of-lhe-arl approach of using fixed kernel sizes overall.
Concluding, we propose NIFF CNNs, tool with which can learn convolution filters in the frequency domain that translate into infinitely large kernels in the spatial domain. NIFF can efficiently replace common spatial convolutions with clement-wise multiplication in the frequency domain. We analyze the resulting kernels in the spatial domain and observe that they are well localized and mostly quite small (9 x 9). On rigl-resolution datasets, NIFF can perform on-par with or better than spatial convolutions and can leverage the benefit of encoding large spatial context.
References
Shashank Agnihotri, Julia Grabinski, and Margret Keuper. Improving stability during upsampling-on on the importance of spatial context. arXiv preprint arXiu2231.117224 2023.
Sayed Omid Ayat, Mohamed Khalil- Ab Al-Hadi Ab Rahman, and Hamdan Abdellatef. Spectral-basod convolutional neural network without multiple spatial-frequency domain switchings. Newrocompuling, 364: 152-167, 2019.
Ron Bracewell and Peter B Kahn, The fourier transform and its applications. American Journal of Physics, 34(8):712 712, 1966.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning contimuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628 8638, 2021.
Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Н. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 4479 4488. Curran Associates, Inc., 2020. URL http:://ppoeeangg.eerrro..cccccp 1:554166662b0//72226666666222221
pddf
Julian Chibane, Aymen Mir, and Gerard Pons-Moll. Neural unsigned distance fields for implicit function learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21638 21652. Curran Associates, Inc., 2020. URL https: //proceedings. neur htw.eeeddddnnsnennisss........
Taco Cohen and Max Welling. Stecrable cmns. arXiv preprint ar arXi::1612.08/4 2016.
James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, :9(90):297 301, 1965.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convo- lutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in enns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 11963 11975, June 2022.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12124 12134, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghann, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint Xix:20000.. 11929, 2020.
George H Dunteman. Principal components analysis, volume 69. Sage, 1989.
Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
D. Forsyth and J. Ponce. Computer Vision: A Modern Approach. An Alan R. Apt book. Prentice Hall, 2003. ISBN 9780130851987. URL htttps:/bbooks wowessgggooge..o w.t.
Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. Frequencyloweut pooling plug & play against catastrophic overfitting. In European Conference Computer Vision, 2022. URL https:
//axxiv.org/aaaa www.iv.orggbbb///2/// 220/ 22091004941111
Julia Grabinski, Janis Keuper, and Margret Keuper. Fix your downsampling asap! be natively more robust via aliasing and spectral artifact free pooling. arXiv preprint arXiv:2807.0003 2023.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https:://penreview..rr
Bochen Guan, Jinnian Zhang, William A Sethares, Richard Kijowski, and Fang Liu. Specnet: spectral domain convolutional neural network. arXiv preprint arXie: 10915, 2019.
Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. arXiv preprint arXiv:2209.0883 085775 2022.
David Ha, Andrew Dai, and Quoc V Lc. Hypernetworks. arXiv preprint wxXiv:169.099100 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 1314-1324, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Steffen Jung and Margret Keuper. Spectral distribution aware image generation. In Proceedings of the AAAI conference arlificial intelligence, volume 35, pp. 1734-1777 1742, 2021.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pochenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. Xiv preprint xrXic:2207.36620 2022a.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 10012 10022, 2021.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A con- vnct for the 2020s. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022b.
Jovita Lukasik, Paul Gavrikov, Janis Keuper, and Margret Keuper. Improving native enn robustness with filter frequency regularization. Transactions on Machine Learning Research, 2023.
Tianyu Ma, Adrian V Dalca, and Mert R Sabuncu. Hyper-convolution networks for biomedical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1933-1942, 2022.
Tianyu Ma, Alan Q. Wang, Adrian V. Dalca, and Mert R. Sabuncu. Hyper-convolttions via implicit kernels for medical image analysis. Medical Image Analysis, 86: 102796, 2023. ISSN 1361-8415. doi: https:// doi.org/ 10. 10..0006...mmeeeeeeee wwtp:......a.......22222...... 1023.9
$136184152300057|
Michael Mathieu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through fits. arXiv preprint 1312 2013.
Hengyue Pan, Yixin Chen, Xin Niu, Wenbo Zhou, and Dongsheng Li. Learning convolutional neural networks in the frequency domain, 2022. URL https://arxivv
Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters improve semantic segmentation by global convolutional network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Silvia L Pintea, Nergis Tömen, Stanley F Goes, Marco Loog, and Jan C van Gemert. Resolution learning in deep convolutional networks using scale-space theory. IEEE Transactions on Image Processing, 30: 8342-8353, 2021.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28043-2042 28078. PMLR, 2023.
Harry Pratt, Bryan Williams, Frans Coenen, and Yalin Zheng. Fenn: Fourier convolutional neural networks. In Michelangelo Ceci, Jaakko Hollmén, Ljupëo Todorovski, Celine Vens, and Sašo Džeroski (eds.), Ma- chine Learning and Knowledge Discovery in Databases, pp. 786 798, Cham, 2017. Springer International Publishing. ISBN 978-3-319-7124999
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. In A. Beygekimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL https://openrevier: wttprrrriiieneee/ouuunureeee hett/oorumidddKMMnsssase
David W. Romero, Robert-Jan Bruintjes, Jakub Mikolaj Tomezak, Erik J Bekkers, Mark Hoogendoorn, and Jan van Gemert. Flexconv: Continuous kernel convolutions with diffcrentiable kernel sizes. In International Conference Learning Represenlations, 2022a. URL https://openrrvvier www.:frufffrr
id=3jooF27-WWy
David W. Romero, Anna Kuzina, Erik J Bekkers, Jakub Mikolaj Tomezak, and Mark Hoogendoorn. CKConv: Continuous kernel convolution for sequential data. In International Conference Learning Representa- tions, 2022b. URL httpp:://openrevier http://eweeeeeeeeeeeereeerr/aaaa
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenctv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp.
2018.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstcin. Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Н. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, Pp.
7462-7473. Curran Associates, Inc., 2020. URL https://proceeedings http://a..c///pppr///////////// Trroopaaoedfoocaaaac4444444ee000
Ivan Sosnovik, Michal Samaja, and Arnold Smeulders. Scale-equivariant steerable networks. ar preprinl arXiv:1910.11093 2019.
Nergis Tomen and Jan C van Gemert. Spectral leakage and rethinking the kernel size in cmns. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5138-5147, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers distillation through attention. In International conference on machine learning, pp. 10347 10357. PMLR, 2021.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher Pal. Deep complex networks. International Conference on Learning Representations, 2018. URL http::::///ppereee...eeeeeeeeeee
H1T2hmZAb
Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, and Yann LeCun. Fast convolutional nets with fbfft: gpu performance evaluation. arXiv preprint arXiv:1412. 7580, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Zelong Wang, Qiang Lan, Dafei Huang, and Mei Wen. Combining fft and spectral-pooling for efficient convolution neural network model. In 2016 2nd International Conference on Artificial Intelligence and Industrial Engineering (AIIE 2016), pp. 203-206. Atlantis Press, 2016.
Thomio Watanabe and Denis F Wolf. Image classification in frequency domain with 2srclu: second harmonics superposition activation function. Applied Soft Compuling, 112:107851, 2021.
Shmuel Winograd. On computing the discrete fourier transform. Mathematics of computation, 32(141): 175 175-199, 1978.
Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. Advances in Neural Information Processing Systems, 32, 2019.
16

---

Published in Transactions on Machine Learning Research (05/2024)
As large as it gets - Studying Infinitely Large Convolutions via Neural Implicit Frequency Filter
Supplementary Material
In the following, we provide additional information and details that accompany the main paper:
A Kernel Mass Evaluation
In this section, we evaluate the kernel mass ratio for more ResNet models trained on ImageNet-1k (Figure Al) and different network architectures trained on ImageNet-100 (Figure A2). The networks show similar behavior already observed the main paper, all models predominately learn small, well-localized kernels regardless of the potential learn much larger kernels. However, the smaller ResNet-18 model learns larger kernels than the ResNet-50 or ResNet-101 the second layer. For ImageNet-1000 MobileNet-v2 does not learn large kernels observed for ImageNet-1k. Further, ResNet-50 trained ImageNet-100 seems to learn larger kernels in the second layers compared to the ResNet-50 trained on ImageNet- Ik (Figure 4
ResNet-18
ResNet-101
1.0
1.0
kernel mass ratio
0.8
kernel mass ratio
0.8
0.6
0.6
Layer (max resolution 56x56)
Layer (max resolution 56x56)
0.4
Layer (max resolution 56x56)
0.4
Layer (max resolution 56x56)
Layer (max resolution 28x28)
Layer (max resolution 28x28)
0.2
Layer 4 (max resolution 14x14)
0.2
Layer (max resolution 14x14)
effective kernel size 9x9
effective kernel size 9x9
0.0
0.0
10
20
30
40
10
20
30
40
50
kernel size
kernel size
Figure A1: Effectivve kernel size evaluation on ImageNet-1k for further ResNet models. We plot the average ratio of the entire kernel mass contained within the limited spatial kernel size, where the x-axis denotes the width and height of the squared kernels. The layers are summarised follows: Layer 1 encoding 56 x 56, Layer 28 28, Layer 14 14 and Layer 4 47x7.
Non-square kernels Following Romero et al. (202aa), we analyse not only square shape kernels, which are typically applied in CNNs but also rectangular shape kernels. Thus, we fit 2D Gaussian to kernels learned using our NIFF and compare the variance given by ag and σy in the x- and y- direction. In detail, we build the ratio between ax and ay of the Gaussian. To aggregate over all kernels within one layer, we plot the mean and standard deviation of these ratios in Figure A3 and Figure A4 The mean over all kernels within layer is near to ratio of one indicating that most kernels exhibit square-shapes. For some layers (the last layer for ResNet-50 and ResNet-101 on ImageNet-lk and ResNet-18 and ResNet-50 ImageNet-100) the variance is quite high, indicating that and dg differ and non-square, rectangle kernels are Iearned. Not. that the learned kernels by Romero et al. (2022a) are parameterized by a Siren Sitzmann el al. (2020) leading to more wave-like, smooth kernels. contrast, we learn the kernels in the frequency domain which could be wave-like, but are mostly not wave-like as shown Figure A22 and Figuure A23 Therefore, the measured standard deviations in and y direction should be understood as a kernel mask as argued in Romero et al. (002aa). They merely indicate the rough spatial distribution of filter weights.

---

Published in Transactions on Machine Learning Research (05/2024)
B Filter Visualization
B.1 Principle Component Analysis
The Principle Component Analysis, short PCA, is typically a dimensionality reduction method. The goal of PCA is to maintain most information of a datasct while transforming it into a smaller onc. The first principle component explains the most variance of the data, thus representing the majority. The second principle component explains the next highest variance while being orthogonal the first. For more details the PCA of the learned kernels to visualize PCA, we refer to Dunteman (1989). For analysis, reduction property of the PCA to simplify the the predominate structure. Hence, we use the dimensionality visualization of the kernels. We also provide images of the original kernels in Figure A22 and A23
B.2 Spatial Kernels
In this section, we show the PCA evaluation of the learned spatial kernels by the NIFFs for additional architectures and datasets (ImageNet-100).. The results similar to the ones in the paper (Figure B Figure [ and Figure 7 The learned filters in the spatial domain are well-localized and relatively small compared to the size they could actually learn. This holds true for different archilectures on ImageNet-1k
(Figure A5 A6 and A7 as well as for ImageNet-100 (Figure A9 A10 and A12
Further, we show a grid of the actually learned filter in the spatial domain in Figure
A22 and
A23
The learned spatial filters on CIFAR-10 are shown in Figure A13
Similarly, to the results shown in Figure The network learns well-localized, small filters n the spatial domain. Yet, for the feature map size of 8 in Layer 2 the network uses significantly more than the standard kernel size of 3 x 3.
B.3 NIFF multiplication weights
Moreover, we analyze the learned element-wise multiplication weights for the real and imaginary parts of different models trained on ImageNet-Ik in the frequency domain. Figures A14 A15 show the PCA per layer for the learned element-wise multiplication weights for ResNet-50, DenseNet- 121, ConvNeXt-tiny and MobileNet-V2 respectively. For ResNet-50 and ConvNext-tiny, it seems if the networks focus in the first layer on the middle-frequennc spectra and in the later layers more on the high-frequency spectra. The multiplication weights learned for MobileNet- V2 (Figure A16) focus in the first layer on low- frequency information in the second layer on high-freque figh-frequency information and in the third layer again on low-frequency information. The DenseNet-121 (Figure A17) Iearns high-frequency information prior in the first two layers and low-frequency information predominately in the later, third layer. Hence, a general claim for different models and their learned multiplication weights in the frequency domain can be derived from our empirical analysis. Still for all networks, the imaginary part scems to be less important for these networks and thus the learned structures are less complex. This might be owed to the fact that with increased sparsity through the activation function in the network, the network favours cosine structures (structures with a peak in the center) over sine structures.
A16 and A17
C Performance Evaluation
ImageNet--100 We report the accuracy our NIFF CNNs could achieve on ImageNet-110 and the number of learnable parameters in Table A1 The trend is similar to ImageNet-1k, the larger models benefit from NIFF while the lightweight models do not much. In addition to the models considered in the main paper, we additionally evaluated MobilcNet-v3 Howard et al. (2019). We observe that both the bascling model and the NIFF version have comparably low accuracy. For Mobile the training pipeline is usually highly optimized for best performance. The data augmentation scheme from Liu et al. (2022b), that we employ for all trainings to achieve comparable results, does not seem have a beneficial effect here, neither on ImageNet-1k (for Sandler et al. (2018). Table 2b nor on ImageNet- (for Sandler al. (2018): Howard et al. 2019)) Table A1).
2

---

Published in Transactions on Machine Learning Research (05/2024)
Table Al: Performance evaluation of top and test accuracy on ImageNet-100 for different network architectures. We used the standard training parameter for each network architecture and stayed consistent with these for each architecture respectively. For the bigger networks like ResNet and DenseNet, which include 2D convolutions, we split into depth-wise and 1x1 convolution as described in Section [3. to reduce the number of parameters, for faster training. For all models, our NIFF CNNs perform slightly better, even with reduced number of parameters.
Name
# Parameters
Accã1
Acc@5
ConvNeXt-tiny Liu et al. (2022b)
27.897.028
91.70
98.32
NIFF (ours)
08.231.684
92.00
98.42
ResNet-18 He et al. (2016)
11.227.812
87.52
97.50
NIFF 2D conv (ours)
20.665.62)
86.42
97.08
ResNet-18 separated cony NIFF (ours)
700
86.52
97.20
3.198.660
86.52
97.14
ResNet-50 He et al. (0116)
23.77 712.912
89.88
98.22
NIFF 2D conv (ours)
31.934.225
90.08
98.06
ResNet-50 separated conv NIFF (ours)
16.431.588
89.78
98.18
16.760.900
89.98
98.44
ResNet-101 He et al. (0066)
42..05000
90.54
98.14
NIFF 2D conv (ours)
60.954.180
89.94
98.14
ResNet-101 separated conv NIFF (ours)
26.549.988
90.20
98.36
27.343.332
90.54
98.38
DenseNet-121 Huang ct al. (2017)
7.056.356
90.06
98.20
NIFF 2D conv (ours)
9.197.252
89.66
98.32
DesNett-111 separated conv NIFF (ours)
5.222.628
89.94
98.08
5.237.012
90.24
98.18
MobileNet-v2 Sandler et al.
20188
2.351.972
84.06
96.52
NIFF (ours)
2.359.660
85.46
96.70
MobileNet-v3 Howard et al. (0199
1.620.356
79.84
94.76
NIFF (ours)
1.617.748
78.90
95.40
CIFAR-10 Although NIFF CNNs can perform on par with the respective baseline on high-resolution datasets, their performance is limited on low-resolution dataset. Table A2 shows the results on CIFAR- 10 with different architectures. Unfortunately, our NIFF CNNs lose around 1 to 3 % points compared to the baseline models. This can be addressed to our previous observation: The networks trained on CIFAR-10 do only use a small amount of the potential kernel size NIFF provides being big the kernels of the bascline model (3 3).
D Circular vs Linear Convolution
Our NIFF as proposed above performs a circular convolution, which allows us to directly apply the convo- lution theorem and execute it as multiplication in the frequency domain. However, standard convolutions in CNNs are finite lincar convolutions. A circular convolution can mimic a lincar convolution when zero- padding a signal with length M and a kernel with length K length S M + K Winograd (1978). Thus, to ablate circular versus finite linear convolutions, the input featuremaps with size N x N
3

---

Published in Transactions on Machine Learning Research (05/2024)
Table A2: Performance evaluation of different networks trained on CIFAR-10 and the number of learnable hyperparameters for each network. To be comparable between all models and architecture changes we used the samne training schedule for all of them. One can see that NIFF CNNs perform slightly better with a ConvNeXt Liu et al. (002bb) backbone. However, for other architectures, it performs slightly worse
Method
# Parameters Top Acc
ConvNeXt-tinn Liu et al. (2022b)
6.376.460
90.37
NIFF (ours)
6.305.746
91.48
ResNet-18 18 He et al. 2016
11.173.962
92.74
NIFF 2D conv (ours)
20.613.546
92.66
ResNet-18 separated cony NIFF (ours)
2.810.341
90.18
1.932.432
90.63
ResNet-50 He et al. 2016
23.520.845
93.75
NIFF 2D conv (ours)
31.743.914
93.39
ResNet-50 separated conv
.989
92.13
NIFF (ours)
15.491.600
93.11
DenseNet-121 Huang et al. (2017) NIFF 2D conv (ours)
6.956.426
93.93
9.099.098
92.47
DenseNet-121 121 separated conv
5.121.189
92.00
NIFF (ours)
5.555.856
92.49
MobileNet-v2 Sandler et al.
22018
2.236.682
94.51
NIFF (ours)
2.593.760
94.03
MobileNet-v3 Howard et al.
2019
1.528.106
86.28
NIFF (ours)
1.526.466
86.60
zero-padded to 2N x 2N. For both, the linear and the circular case, NIFF learns filters with the original size N x N of the fcaturemap. To mimic lincar filters, the learned filters by our NIFF are transformed into the spatial domain and zero-padded similarly to the input featuremaps to 2N x 2N. Afterwards, they are transformed back into the frequency domain and the point-wise multiplication is executed. Note that this is not efficient and just serves the academic purpose of verifying whether any accuracy is lost when replacing linear convolutions by circular ones in our approach. However, the resulting networks experience performance drop compared to the baseline and our NIFF as shown in Table A3 and Table A4 We hy- pothesize that this drop in performance results from the enforcement of really large kernels. The additional padding mimics linear finite convolutions that are as big as the featuremaps. Related work has shown that larger context can improve model performance Ding et al. (2022); Liu et al. (2022a). Still, there is a limit to which extent this holds as with large kernels artifacts may arise Tomen & van Gemert (2021). Thus, enforcing kernels large as the featuremaps seems be not beneficial as shown by our quantitative results. Another explanation for the drop in accuracy could be the introduction of sinc interpolation artifacts into the padded and transformed featuremaps and kernels. The padding is formally a point-wise multiplication with a box-function in the spatial domain. Thus, sine-interpolation artifacts in the frequency domain can arise. Figure A20 and Figure A2T show that the learned spatial kernels are larger than the learned kernel when we apply a circular convolution with our NIFF. While the first and second layers still learn relatively small filters compared to the actual size they could learn, the third and fourth layers make use of the larger kernels. Since these results come with a significant drop accuracy, we should however be careful when interpreting these results.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Table A3: Evaluation of top 1 and top accuracies on ImageNet-100 for networks learning filters with NIFF but afterwards those are padded in the spatial domain and transformed back into the frequency domain to mimic linear convolutions. All ResNet archilectures and DenseNet-121 were trained with separated depth- and Ixl convolutions due to efficiency reasons. All models using finite linear convolutions perform significantly worse than the baseline and our NIFF which applies circular convolutions. This observation is consistent with low-resolution data like CIFAR-10.
Name
Acc@1 1
Acc@
ConvNeXt-tiny Liu (002b
91.70
98.32
NIFF (ours)
92.00
98.42
NIFF linear
83.00
95.36
ResNet-18 separated conv
86.52
97. 97.20
NIFF (ours)
86.52
97.14
NIFF linear
81.90
95.42
ResNet-50 separated conv
89.78
98.18
NIFF (ours)
89.98
98.44
NIFF lincar
86.76
97.16
ResNet-101 separated conv
90.20
98.36
NIFF (ours)
90.54
98.38
NIFF linear
86.70
97.06
DesNet-121 scparated conv
89.94
98.08
NIFF (ours)
90.24
98.18
NIFF linear
81.40
95.52
MobileNet-v2 Sandler et al. (2018)
84.06
96.52
NIFF (ours)
85.46
96.70
NIFF lincar
73.90
93.16
E
Runtime
As discussed in the computing costs section of the main paper, our approach is slower than the current implementation with spatial convolutions due to the repetitive use of FFT and IFFT. However, when com- paring the number of FLOPs needed to compule convolutions with kernel sizes as big as the Teaturemaps our NIFF approach, NIFF requires significantly fewer FLOPs, especially with increased featuremap size. Figure g shows that most of the FLOPs for NIFF result from the additional FFT and IFFT operation. Still, require much fewer FLOPs than large spatial convolutions.
Moreover, we evaluate the runtime per epoch for each model on CIFAR-10 (Table A5 and ImageNet-1O0 100 (Table A6 and compare it to the standard spatial 3x3 convolution, which has a much smaller spatial context than our NIFF as well as spatial convolutions which are as large as the featuremaps. This would be comparable to our NIFF. Obviously, small spatial kernels (3 3) are much faster than larger kernels like NIFF or large spatial kernels. However, NIFF is much faster than the large spatial kernels during training, Especially on high-resolution datasets like ImageNet-100 our NIFF is over four times faster ResNet-50 and over three times faster ConvNeXt-tiny compared to the large convolution in the spatial domain.
In general, we want to emphasize that our NIFF models still learn infinite large kernels while all kernels in the saltial domain are limited to the set kernel size. If one would like to learn a 2D convolution in the spatial domain with an image g size N x N and filters with the same size N x this would be in O(N4) whercas using FFT and pointwise multiplication (Equation would result in O(N

---

Published in Transactions on Machine Learning Research (05/2024)
Table A4: Evaluation of top on CIFAR-10 for networks learning filters with our NIFF but afterwards those padded in the spatial domain and transformed back into the frequency domain to mimic linear, non-cireular convolutions. All models using finite linear convolutions perform significantly worse than the baseline and our NIFF. This observation is consistent with high-resolution data like ImageNet-1000 100.
Method
Top 1 Acc
ConyNeXt-tiny Liu et al. (2022b)
90.37
NIFF (ours)
91.48
NIFF linear
84.30
ResNet-18 He et al. (0016)
92.74
NIFF full (ours)
92.66
NIFF full linear
85.10
ResNet-18 scparated conv
90.18
NIFF (ours)
90.63
NIFF linear
83. 11
ResNet-50 He et al. (0066)
93.75
NTFF full (ours)
93.39
NIFF full linear
88.22
ResNet-50 separated conv
92.13
NIFF (ours)
93.1 33.11
NIFF linear
87.54
DenseNett-22 121 Huang al, (2017)
93.93
NIFF full (ours)
92.47
NIFF full linear
85.73
DenseNet- separated conv
92.00
NIFF (ours)
92.49
NIFF linear
79.95
MobileNet-v2 Sandler et al. (2018)
94.51
NIFF (ours)
94.03
NIFF linear
93.21
Table A5: Average training time per epoch in seconds and standard deviation on one NVIDIA Titan of NIFF compared standard spatial convolutions 3x3 or 7x7 and maximal larger spatial convolutions on CIFAR-10.
Name
Baseline /x7
Spatial Conv
NIFF (ours)
featuremap sized
ConvNeXt--inn Liu et al.
2022bb
68.31 + 0.62
108.97 + 0.25
96.48 + 1.97
ResNet-18 He et al. [2016
8.57 = 0.20
22.75 = 0.28
17.87 + 0.54
ResNet-50 He el. al. (2016
7.98 = 0.10
37.05 + 0.48
27.36 +0.16
DenseNett 21 Huang et al.
(2017
26.36 + 1.32
67.11 = 0.25
84.51 + 3.71
MobileNet-v2 Sandler et al.
12018
22.50 + 0.26
143.47 0.20
83.41 + 4.78
F Ablation on more modules
We show that our NIFF can be combined with other frequency modules achieve networks that operate mostly in the frequency domain. Hence, the number of transformations can be reduced. Table Ag demon-

---

Published in Transactions on Machine Learning Research (05/2024)
Table A6: Average training time per epoch in seconds and standard deviation on four NVIDIA A100 of NIFF compared to standard spatial convolutions (3x3 7x7) and maximal larger spatial convolutions ImageNet-100.
Name
Basclinc 3x3/7x7
Spatial Conv Teaturemap sized
NIFF (ours)
ConvNeeXt-ting tiny Liu al.
2022bb
92.19 = 4.21
487.89 + 3.54
149.70 = 1.32
ResNet-118 He et al.
2016
31.99 + 0.71
608.13 + 22.53
89.00 = 0.60
ResNet-50 He et al.
2016
85.51 = 0.22
951.43 + 6.09
204.82 + 0.33
ResNet-100
He et a..
[2016
152.39 + 5.07
1392.21 + 47.91
349.83 = 2.44
DenseNet-121 Huang et. al.
[2017]
128.95 + 1.64
10188.15 + 28.17
408.08 + 2.20
MobilcNet-v2 Sandler et al.
[0118]
32.64 = 0.25
856.84 + 4.35
100.75 + 0.15
Table A7: Average inference time in seconds and standard deviation on one NVIDIA Titan V of NIFF compared to standard spatial convolutions (3x3 or 7x7) and maximal larger spatial convolutions on the full CIFAR-10 validation set.
Name
Baseline
Spatial Conv
NIFF (ours)
3x3/7x7
featuremap sized
Con-NeXt-tiny Liu et al.
(2022b)
2.35 0.05
4.06 + 0.02
6.32 0.14
ResNet-18 He al. (2016
2.69 0.10
3.29 0.08
3.14 + 0.54
ResNet-50 He al.
(2016)
3.09 + 0.13
5.05 + 0.45
5.26 0.06
DenseNet- 12 Huang al. 2017
3.40 = 0.05
5.27 = 0.02
6.50 0.14
MobilcNet-v2 Sandler et al.
12018)
2.93 + 0.06
8.68 + 0.43
7.02 + 0.04
strates that adding the downsampling layer Grabinski et al. 2022 2023 or the last average pooling and the fully connected layer also yields good results. Also combining all of them, NIFF, FLC Pooling Grabinski et al. (2022) and the Average Pooling plus Fully connected layer performs quite well. Also, incorporating the ComplexBatchNormn Trabelsi al. (2018) leads to a drop accuracy by roughly 15%. We also tried incorporate the non-linearity into the frequency domain, but we were not. able to achieve much better results than by removing it fully.
G Ablation on Padding
We evaluate NIFF when the featuremaps are padded with different kinds of padding methods. Due to the padding of the featuremaps and the cropping after the application of our NIFF, possible artifacts be mitigated. The padding is applied around the fcaturemaps before transforming them into the frequency domain. The padding size is as large as the original featuremap. After the application of our NIFF, the featuremaps are transformed back into the spatial domain and cropped to their original size. The resulting networks experience similar performance as our baseline NIFF as shown in [ablle A10 Figure A18 and Figure A19 show the learned spatial kernels when only the featuremaps are padded. The learned spatial kernels still relatively small and well-localized.
NIFF's architecture
In the following, we describe the architecture used for our NIFFs for each backbone network architecture. Note that the size of the NIFF is adjusted to the size of the bascline network as well the complexity of the classification task.

---

Published in Transactions on Machine Learning Research (05/2024)
Table A8: Average inference time in seconds and standard deviation on four NVIDIA A100 of NIFF compared to standard 3x3 full 7x7 depth-wise spatial convolutions and maximal larger spatial convolutions the full ImageNet-100 validation set.
Name
Baseline
Spatial Conv
NIFF (ours)
3x3/7x7
foaturemap sized
ConvNeXt-tinn Liu al.
2022b)
4.86 + 0.13
17.07 = 0.19
6.06 + 0.08
ResNet-18 He al. 2016
4.53 + 0.52
24.57 0.21
4.52 + 0.18
ResNet-50 He al.
2016
5.43 = 0.30
42.33 0.21
10.35 = 0.14
ResNet-100 He et al.
(2016)
7.35 + 0.17
54.85 = 0.25
16.82 = 0.10
DenseNet-121 Huang et al.
2017
5.12 + 0.07
104.36 = 0.40
12.04 = 0.18
MobileNet-v2 Sandler et al.
2018
4.27 + 0.13
28.97 = 0.04
6.61 0.16
Table A9: Comparison on CIFAR-10 of NIFF MobileNet-v2 Sandler et al. 20088
incorporating more modules
besides our NIFF into the frequency domain.
NIFF
FLC Pooling
2022
Complex BatchNorm
2018
AveragePooling + FC
Accuracy
x
X
94.03
X
93.61
X
78.60
X
X
93.82
79.83
V
x
93.27
X
V
73.90
V
55.81
Low-resolution task All networks trained on CIFAR-10 incorporate the same NIFF architecture. The NIFF consists of two stacked 1 convolutions with ReLU activation function in between. The 1 x convolution receives as imput two channels, which encode the x and y coordinate as described in Figure 2 The x 1 convolution expands these two channels to 32 channels. From these 32 channels, the next x convolution maps the 32 channels to the desired number of point-wise multiplication weights.
High-resolution task For the networks trained on ImageNet- and ImageNet- 1k the size of the neural implicit function predict the NIFF kept the same for each architecture respectively, while the size of the neural implicit function is adjusted to the network architecture to achieve approximately the same number of trainable parameters. Hence, the lightweight. MobileNet-v2 model [Sandler et. al. (2018) and the small DensNet-121 Huang et al. (017)) incorporate a smaller light-weight neural implicit function to predict the NIFF, while larger models like ResNet He et al. (2016) or ConvNeXI-tiny Liu et al. (202bb) incorporate a larger neural implicit function. For simplicity, we define two NIFF architectures. One for the large models and one for the smaller, lightweight models.
For the smaller,, lightweight models, the neural implicit function consists of three stacked x convolutions with one SiLU activation after the first one and one after the second one. The dimensions for the three 1 convolutions are as follows. We start with two channels and expand to eight channels. From these eight channels, the second 1 x 1 convolution suppresses the channels down to four. Afterwards, the last 1 x convolution maps these four channels to the desired number of point-wise multiplication weights.
For the larger models, we used four layers within the neural implicit function for NIFF. The structure is similar to all NIFFs between each convolution a SiLU activation function is applied. The dimensions for the four layers follows. First. from two to 16 channels, secondly from 16 to 128 channels and
8

---

Published in Transactions on Machine Learning Research (05/2024)
Table A10: Evaluation of top 1 and top accuracies on ImageNet-100 for ResNet- 18 with different kinds of padding.
Name
Acc@1
Acc@5
ResNet-18 baseline
87.52
97.50
NIFF (ours)
86.52
97.14
NIFF zero padding
87.00
97.54
NIFF reflect padding
86.64
97.24
NIFF circular padding
87.06
97.34
afterwards suppressed down from 128 to 32 channels. The last x convolution maps these 32 channels to the desired number of point-wise multiplication weights.
We show that the smaller NIFF size for the lightweight models does not influence the resulting performance. Thus, we train a lightweight. MobileNet-v2 with larger NTFFs (similar size as the larger models). The results are presented in Table [A1] We can see that the network does not benefit from the larger NIFF size. Hence, assume that keeping the smaller NIFFs for the smaller, lightweight models can achieve good trade-off between the number of learnable parameters and performance.
Table A11: Evaluation of top and top accuracies on ImageNeet110 for different. NIFF sizes for the lightweight MobileNet-v2 Sandler al. (2018).
Name
Acc@1
Accê5
MobilcNet-v2 bascline
84.94
96.28
small NIFF
83.72
96.40
big NIFF
83.82
96.32
Low-resolution task For all models trained on CIFAR-10 the NIFF architectures is kept the same. The neural implicit function consists of two stacked convolutions with one ReLU activation in between. The dimensions for the two 1 x 1 convolutions are as follows. We start with two channels and expand to 32 channels. The sccond 1 x 1 convolution maps these 32 channels to the desired number of point-wise multiplication weights.
Ablation on Separated Convolution Further, we ablate our desigu choices to separated depth-wise and 1 x convolutions instead of full convolutions for efficiency. Hence, we train all ResNet and DenseNet networks with additionally separated convolutions (separated in depth-wise and 1 x 1 convolution) as well as our NIFF as full convolution. Tables 2 A1 and A2 show that using separated convolutions in the spatial domain performs slightly worse than the baseline but also reduces the amount of learnable parameters similarly our NIFF. Using full convolutions in our NIFF leads to an increase in accuracy but also an increased amount of learnable parameters. Hence, we see a clear (rade-off between number of learnable parameters and accuracy.
Training Details
ImageNet. The training parameters and data preprocessing are kept the same for ImageNet-1k and ImageNet-100. For the training of each network architecture, we used the data preprocessing as well the general training pipeline provided by Liu et al. (002b)). The training parameters for each individual

---

Published in Transactions on Machine Learning Research (05/2024)
network are taken from the original papers provided by the authors ResNet He et al. 2016
DenscNeet121
Huang et al. (2177) ConvNeXi-tiny Liu et al. (2022b) and MobileNet-v2 Sandler et al. (011
CIFAR-10. For CIFAR-10 we used the same training parameter for all networks. We trained each network for 150 epochs with batch size of 256 and a cosine learning rate schedule with learning rate of 0.02. we set the momentum to 0.9 and weight decay to 0.002. The loss calculated via LabelSmoothingLoss with label smoothing of 0.1 and an optimizer, we use Stochastic Gradient Descent. (SGD).
For data preprocessing, we used zero padding by four and cropping back to 32 32 and horizontal flip, well as normalizing with mean and standard deviation.
Computing Infrastructure For training our models and the baseline we use NVIDIA Titan V and NVIDIA A100 GPUs. For the training on low-rcsolution data (CIFAR-10). We used one NVIDIA Titan V, depending on the model architecture and the convolution used (baseline, NIFF or large convolution) the training took between 15 minutes and 90 minutes. For the training on high-resolution data (ImageNVet-000 and 1k) we used four NVIDIA A100 in parallel. The training time depends on the used model architecture and varies if we used the full ImageNet- 1k dataset or only ImageNet-100. The training time for ImageNet- 1k varies between one day and one hour and ten days and nine hours for ImageNet- 100 betweer 93 minutes and one day cight hours dependent on the model architocture and the number of epochs for training.
J Convolution Theorem
Following, we demonstrate the proof of the convolution theorem. For more details, please refer to (for example) Bracewell Kahn (1966): Forsyth Ponce (2003).
As stated in Equation 1 in the main paper make use of the convolution theorem Bracewell & Kahn 1966)
Forsyth & Ponce (2003) which states that circular convolution, denoted by between signal g and filter k(x) in the spatial domain can be equivalently represented by woint-wise multiplication, denoted by Ø, of these two signals in the frequency domain, by computing their Fourier Transform, denoted by the function F():
=)g F F(k)
(7)
with
F(g(x)) = G(u) / 30
g(z)e j22uu dx
(8)
To show that this holds, we first show that the Fourier transformation as a system has specific properties when the signal shifted. we shift a signal/function g(x) by a in the spatial domain expressed by g(x a) this results in a linear phase shift in the Fourier domain:
Flg(z -
a)) a)) F(g(z')) = - g(aa)e--2aull -
j2mulz' +aa) dx!
(9)
where e (-2mu(z'aa) 52ruu f f2our' and e --zuu is constant, such that
F(g(a - -)) = = - j2mua G(u)
(10)
Using the shift property of the Fourier transform we now prove the convolution theorem. The continuous convolution is defined as follows:
g(x) @ k(x) = /
g(z)k(y- - x)dæ
(11)
10

---

Published in Transactions on Machine Learning Research (05/2024)
The Fourier transformation of g(x) @ k(x)) is defined by:
s(z)k(y r)dx e- -yanuy
(12)
By reversing the order of the integration we get.
[ g(x)
[ kly z)e- 32 -2e-d2 dy
de
(13)
where we can pull out g(z). Given the shift property, the inner integration can be defined by:
f kly x)e Januy F(k(y =x)) -jarus K(u)
(14)
such that
g(0)
k(y - z)e toe-jan j2zuy dy de
g(z)e -aauz K ((u)de
(15)
1 g(x)) -2 de 1
K(u)
((u) F(H(())
so that for all spatial frequencies u, we have
Flg Flg)(u) o F(k)(w)
(16)
K
Fast Fourier Transform
The Discrete Fourier Transform (DFT) an input signal f(n) with N samples is defined as
N Σ
F(k)
f(m)e-22nka j2nknNN
(17)
n=D
Executing the DFT directly would take O(N²). Thus Cooley & Tukey (1965 developed the Fast Fourier Transform, short FFT. Which builds upon a divide and concur strategy and reduces the runtime down to ()NlogN).
They used the inherent symmetry which results from the period nature of the transformed signal. To give intuition for this inherent symmetry lets explore what happens if we shift by N:
N-1
F(k: + N)
Σ
f(n)e (N)(NN)
N-1
Σ
f(n)e -22m f- J22kknNN
(
(18)
N-1
Σ
f(n)e -jrkn/NN
=
n=0
11

---

Published in Transactions on Machine Learning Research (05/2024)
efRn 1 for any integer Thus see that
F(k + N) F())
(19)
and also
F(k F(k+ +
iN) F(k)
(20)
for any integer i holds.
Given this symmetry, Cooley Tukey 1965) developed an algorithm which divides the DFT into smaller parts such that the DFT can be solved divide and concur. Following we rearrange the DFT into two parts:
F(k) E f(n)e 22
N/2-1
[(2m)e j2nk2m/N
m=0
N/2-1
Σ
[(2m + 1)e
j2nk(2m+1/N
(21)
m
N/2-1
Σ
f(2m))e-2nkkm((N7) (2nkm/(N/22)
N/2-1
-2sk/
Σ
f(2m 1)e f2 (N/2)
Each part represents the even numbeeeee and odd-numbered values respectively. However, the runtime is still the same as cach term consist of ONN22)N computations so in total still O(N²).
Luckily, this division into two parts be continued in each part again. Hence, the range of k is 0 S k < N while is in the range of 0 S < M where M N/2. Thus, solving the problem only takes half of the computations as before, O(N2) becomes ((M²) where M is half the size of N. As long as M is even-valued, we can apply divide the problem in even smaller parts, applying the divide and concur strategy which in an recursive implementation takes only ONNoogN).
L Code Base
Implementation code for our NIFF CNNs is provided at: https://anonymouus .open seceemmsceeneer) NIFF NIFF1528anna 10FF1158aanoymuus and will be made publicly available upon acceptance.
12

---

Published in Transactions on Machine Learning Research (05/2024)
ResNet-18
ResNet-50
1.0
1.0
kernel mass ratio
0.8
kernel mass ratio
0.8
0.6
0.6
Layer (max resolution 56x56)
Layer (max resolution 56x56)
0.4
Layer 2 (max resolution 56x56)
0.4
Layer (max resolution 56x56)
Layer (max resolution 28x28)
Layer (max resolution 28x28)
0.2
Layer (max resolution 14x14)
0.2
Layer (max resolution 14x14)
effective kernel size 9x9
effective kernel size 9x9
0.0
0.0
10
20
30
10
20
30
40
50
kernel size
kernel size
ResNet-101
ConvNeXt-tiny
1.0
1.0
kemel mass ratio
0.8
kerne mass ratio
0.8
0.6
0.6
Layer (max resolution 56x56)
Layer (max resolution 56x56)
0.4
Layer (max resolution 56x56)
0.4
Layer (max resolution 56x56)
Layer (max resolution 28x28)
Layer (max resolution 28x28)
0.2
Layer 4 (max resolution 14x14)
0.2
Layer 4 (max resolution 14x14)
effective kernel size 9x9
effective kernel size 9x9
0.0
0.0
10
20
30
40
10
20
30
40
50
kernel size
kernel size
DenseNet- 121
MobileNet-v2
1.0
1.0
kernel mass ratio
0.8
kernel mass ratio
0.8
0.6
0.6
resolution 112x112
Layer 1 (max reslution 56x56)
max resolution 56x56
0.4
Layer (max resolution 28x28)
0.4
max resolution 28x28
Layer (max resolution 14x14)
max resolution 14x14
0.2
Layer 4 (max resolution 7x7)
0.2
resolution 7x7
effective kerne size 9x9
effective kernel size 9x9
0.0
0.0
10
20
30
40
50
20
40
60
80
100
kernel size
kernel size
Figure A2: Effective kernel size evaluation on ImageNet-100.. We plot the average ratio of the entire kernel mass contained within the limited spatial kernel sizc, where the x-axis denotes the width and height of the squared kernels. For ResNet and ConvNeXt-tiny each layer encodes one resolution. Thus, the layers could be summarised (Layer encoding 56 56, Layer 2 56, Layer For DenseNet-121 each layer can be summarised similarly, yet the after the first layer the feature maps are already downsampled resulting in the following: Layer 1 encoding 56 56, Layer 2 28 x 28, Layer 3 14 x 14 and Layer 4 7 x 7. However, for MobileNet-v2 the resolution is downsampled within layer.
28 and Layer 4 14 14).
13

---

Published in Transactions on Machine Learning Research (05/2024)
ResNet-18
ResNet-50
ResNet-101
Mart
2.9
1.9
6.54
layer-depth
layer-depth
layer-depth
MobileNet-v2
DenseNet-121
ConvNext-tiny
1.
20-
1.-
82-
resolution
layer-depth
layer-depth
Figure A3: Analysis non-square kernel shapes inspired by Romero al. (2022a) on ImageNet- 1k. We compare the variance d and By in x- and y-direction of Gaussian fitted onto our learned spatial weights, The red dashed line indicates squarc-shaped kernels the variance or and oy equal.
ResNet-18
ResNet-50
ResNet-101
1011 Scan lene)
Stur venvll
Stars mema our al hameh
25d car stlsene
14-
H9lc
leyer-depth
opea
layer-depth
Rye
1043
layer-depth
Spin
MobileNet-v2
DenseNet-121
ConyNeXt- tiny
Soat's WEARE
12-
1.0
654
90-
19
Mprd
layer depth
layth
Figure A4: Analysis of non-square kernel shapes inspired by Romero al (022a. ImageNet-100. We compare the variance and o, in x- and y-direction of Gaussian fitted onto our learned spatial weights. The red dashed line indicates square-shaped kernels as the variance and d, are equal.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V2
V
V4
V5
V6
1.00
Layer 56x56
+
0.75
0.86
0.04
0.03
0.03
0.03
0.01
<0.01
0.50
Layer 2
56x56
0.25
0.78
0.07
0.06
0.04
0.03
<0.01
<0.01
0.00
Layer 3 28x28
-0.25
0.79
0.06
0.05
0.04
0.02
0.02
<0.01
14x14
Layer
-0.5
0.41
0.35
0.17
0.04
0.01
0.01
<0.01
-1.00
Figure A5: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of a ConvNeXi-tiny trained on ImageNet-1k zoomed to 9 x 9. On the left, the maximal filter size for the corresponding layer given. ConvNeXt convolutions are standardly equipped with larger kernel sizes than usual (7 7). However, our analysis reveals that the network barely uses large filters if it gets the opportunity to learn large filters. The learned filters the first and third layer mostly use small (3 x 3), well-localized filters.
Vo
V1
V2
V/
V4
V5
V6
Layer 1
56x56
0.75
0.96
0.02
<0.01
<0.01
<0.01
<0.01
<0.01
0.50
Layer 2
28x28
0.25
0.98
0.02
<0.01
<0.01
<0.01
<0.01
<0.01
0.00
Layer 3 14x14
-0.25
0.97
0.02
<0.01
<0.01
<0.01
<0.01
<0.01
-0.50
Layer 4
7x7
-0.75
0.98
0.01
<0.01
<0.01
<0.01
<0.01
<0.01
-1.00
Figure A6: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of DenseNet-121 trained ImageNet-1k 1k zoomed to 9 9. On the left, the maximal filter size for the corresponding layer is given. We can see that most filters only use well-localized, small kernel size although they could use a much bigger kernel.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V2
3
V4
V5
V6
1.00
128x128
Stage
:
0.78
0.11
0.10
0.01
<0.01
<0.01
<0.01
Stage
56x56
0.50
0.84
0.08
0.05
0.03
<0.01
<0.01
<0.01
0.25
Stage 28x28
0.00
0.72
0.08
0.07
0.06
0.04
0.01
0.01
14x14
Stage
-0.50
0.81
0.05
0.05
0.04
0.02
0.01
<0.01
Stage
7x7
-0.75
0.85
0.09
0.02
0.01
0.01
0.01
<0.01
-1.00
Figure A7: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of MobileNet-v2 trained on ImageNet-1k zoomed to 9 9. On the left, the maximal filter size for the corresponding stage is given. For MobileNet-v2 the feature maps are downsampled within layer, thus the stages are combine by feature maps size rather than the layers. We can see that most filters only use a well-localized, small kernel size although they could use a much bigger kernel.
Vo
V1
V2
V/
V4
V5
V6
1.00
56x56
Layer
0.75
0.62
0.17
0.10
0.04
0.04
0.01
0.01
0.50
Layer 2
56x56
0.25
0.75
0.15
0.04
0.01
0.01
0.01
<0.01
Layer 3 28x28
-0.25
0.90
0.06
0.02
0.01
<0.01
<0.01
<0.01
-0.50
14x14
Layer
-0.7
0.87
0.09
0.02
<0.01
<0.01
<0.01
<0.01
Figure A8: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of ResNet-50 trained ImageNet-1k original size (not zoomed). On the left, the maximal filter size for the corresponding layer is given. We that most filters only use well-localized, small kernel size although they could use much bigger kernel.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V4
V5
V6
1.00
Layer 56x56
0.75
0.74
0.14
0.05
0.03
0.01
0.01
0.01
0.50
Layer 2
56x56
-0.25
0.69
0.18
0.04
0.03
0.02
0.02
<0.01
0.00
Layer 3 28x28
0.77
0.07
0.04
0.03
0.03
0.01
0.01
-0.50
Layer 4
14x14
-0.75
0.82
0.13
0.02
0.01
0.01
<0.01
<0.01
-1.00
Figure A9: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of a ResNet-50 trained on ImageNet-100 zoomed to On the left, the maximal filter size for the corresponding layer given. We can that most filters only use really small kernel size although they could much bigger kernel.
Vo
V1
V2
V4
V5
V6
1.00
Layer 1 56x56
:
0.75
0.88
0.03
0.03
0.03
0.02
<0.01
<0.01
0.50
56x56
Layer
0.25
0.89
0.04
0.02
0.02
0.02
<0.01
<0.01
0.00
Layer 3 28x28
-0.25
0.82
0.11
0.03
0.02
0.02
<0.01
<0.01
-0.50
14x14
Layer
-0.75
0.70
0.24
0.03
0.01
0.01
<0.01
<0.01
Figure A10: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of a ConvNeXt-tiny trained on ImageNet-100 zoomed On the left, the maximal filter size for the corresponding layer given. We can that most filters only use really small kernel size although they could use much bigger kernel.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V/
V4
V5
V6
1.00
Layer 56x56
1:
0.87
0.06
0.02
0.02
0.01
0.01
0.01
0..50
Layer 2
28x28
0.25
0.80
0.09
0.04
0.03
0.02
0.01
0.01
Layer 3
14x14
0.80
0.08
0.05
0.02
0.02
0.01
<0.01
-0.50
Layer 4
7x7
-0.75
0.83
0.11
0.03
0.02
0.01
<0.01
<0.01
Figure A11: PCA basis and explained variance for each basis vector (below) of all spatial filters for each On the left, the maximal filter size for
layer of DenseNet-121 121 trained on ImageNet-100 zoomed the corresponding layer is given. We can that most filters only use well-oocalized, small kernel size although they could use much bigger kernel.
18

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V2
V3
V4
V5
V6
1.00
Stage 128x128
0.75
0.98
0.01
0.01
<0.01
<0.01
<0.01
<0.01
Stage
56x56
0.50
0.90
0.06
0.03
0.01
0.01
0.01
<0.01
0.25
Stage 28x28
0..00
0.78
0.16
0.03
0.01
0.01
<0.01
<0.01
Stage
14x14
0.25
-0.50
0.72
0.14
0.07
0.03
0.01
0.01
0.01
Stage
7x7
-0.75
0.72
0.15
0.09
0.02
0.01
<0.01
<0.01
-1.00
Figure A12: PCA basis and explained variance for each basis vector (below) of all spatial filters for each layer of MobileNet-v2 trained on ImageNet-100 soomed to 9 x 9. On the left, the maximal filter size for the corresponding stage is given. For MobileNet-v2 the feature maps are downsampled within layer, thus the stages are combine by feature maps size rather than the layers. We can see that most filters only use a well-localized, small kernel size although they could use a much bigger kernel.
19

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V2
V3
V4
V5
V6
1.00
32x32
0.75
0.59
0.18
0.12
0.06
0.02
0.01
0.01
16x16
0.50
0.42
0.24 4
0.16
0. .2
0.03
0.02
0.01
0.25
NIFF
8x8
0.000
0.31
0.24
0.21
0.17
0.04
0.03
<0.01
-0.25
4x4
0.38
0.29
0.18
0.10
0.05
<0.01
<0.01
-0.50
Standard
3x3
-07.5
0.42
0.20
0.13
0.13
0.05
0.04
0.02
-1.00
Figure A13: PCA basis and explained variance for each basis vector (below) of all spatial filters for each resolution for the NIFF convolutions of MobileNet-V2 trained on CIFAR-10 as well as the learned filters for the third layer a standard MobileNet-Ve V2 trained on CIFAR-10 (bottom row). On the right, the maximal filter size for the corresponding layer given. We can see that most filters only use a well-localized, small kernel size although they could use much bigger kernels.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V2
V3
Vo
V1
V2
V3
Layer 1
56x56
0.25
Real
Imag
0.20
0.61
0.17
0.10
0.04
0.61
0.17
0.10
0.04
Layer 2
56x56
Real
Imag
0..5
0.10
0.73
0.15
0.05
0.02
0.73
0.15
0.05
0.02
Layer 3
28x28
Real
Imag
0.05
0.90
0.06
0.02
0.01
0.90
0.06
0.02
0.01
0.00
Layer 4 14x14
Imag Baa
Real
-0.05
0.59
0.36
0.04
0.01
0.59
0.36
0.04
0.01
0.10
Figure A14: PCA basis and explained variance for each basis vector (below) of all element-wise multiplication weights for the real and imaginary part in the frequency domain for each layer a ResNet-50 trained on ImageNet-1k: 1k. On the left, the maximal filter size for the corresponding layer given. Right the weights for the real values are given, and on the left are the imaginary values.
Vo
V1
V2
V3
Vo
V1
V2
V3
56x56 Real
Layer
Imag
0.92
0.04
0.03
0.01
0.92
0.04
0.03
0.01
Layer 2
56x56
Imag
Real
0.1
0.84
0.07
0.05
0.02
0.84
0.07
0.05
0.02
Layer Layer 3
28x28
Real
Imag
0.0
0.85
0.07
0.04
0.02
0.85
0.07
0.04
0.02
14x14
Real
Imag
-0.1
0.55
0.32
0.07
0.03
0.55
0.32
0.07
0.03
Figure A15: PCA basis and explained variance for each basis vector (below) of all element-wise multiplication weights for the real and imaginary part in the frequency domain for each layer of ConyNeXt-ting trained on IeageNet-lk. On the left, the maximal filter size for the corresponding layer given. Right the weights for the real values are given, and on the left are the imaginary values.
21

---

Published in Transactions on Machine Learning Research (05/2024)
V3
Vo
V1
V2
V3
112x112
0.2
Real
0.99
0.01
<0.01
<0.01
0.99
0.01
<0.01
<0.01
0.1
56x56
Real
mmaa
0.0
0.84
0.08
0.05
0.03
0.84
0.08
0.05
0.03
28x28 Real
mma
-0.1
0.78
0.11
0.07
0.03
0.78
0.11
0.07
0.03
-.2
14x14 Real
Jmag
-0.3
0.79
0.07
0.05
0.04
0.79
0.07
0.05
0.04
7x7
Real
-0.4
0.73
0.15
0.09
0.01
0.73
0.15
0.09
0.01
Figure A16: PCA basis and explained variance for cach basis vector (below) of all dlement-wise multiplication weights for the real and imaginary part in the frequency domain for each layer of a MobileNet-v2 trained on ImageNet-1k: On the left, the maximal filter size for the corresponding layer given. Right the weights for the real values are given, and on the left are the imaginary values.
Vo
V1
V2
V3
Vo
V1
V2
V/
Layer 1
56x56
Real
Imag
0.2
0.98
0.01
<0.01
<0.01
0.98
0.01
<0.01
<0.01
Layer 2
28x28
Real
gg Imag
0.98
0.01
0.01
<0.01
0.98
0.01
0.01
<0.01
Layer 3
14x14
Real
-0.1
0.98
0.01
<0.01
<0.01
0.98
0.01
<0.01
<0.01
Layer 4
7x7
Real
Imag
0.93
0.04
0.02
<0.01
0.93
0.04
0.02
<0.01
Figure A17: PCA basis and explained variance for each basis vector (below) all element-wise multiplication weights for the real and imaginary part in the frequency domain for cach layer of DenscNet-121 trained Nel- On the left, the maximal filter size for the corresponding layer given. Right the weights for the real values are given, and on the left are the imaginary values.
22

---

Published in Transactions on Machine Learning Research (05/2024)
V
V1
V2
V3
V4
V5
V6
1.00
Layer 1
56x56
0.87
0.10
0.01
0.01
0.00
0.00
0.00
0.50
56x56
Layer
0.92
0.03
0.03
0.01
0.00
0.00
0.00
0.00
Layer Layer 3
28x28
0.78
0.20
0.00
0.00
0.00
0.00
0.00
0.50
14x14
0.70
0.25
0.02
0.02
0.01
0.01
0.00
-1.00
Figure A18: Actual kernels in the spatial domain of ResNet-18 with additional zero padding before our NIFF trained on ImageNet-100 We plot for each kernel the zoomed-in (9 x 9) version below for better visibility. Still, most kernels exhibit well-localized, small spatial kernels.
Vo
V1
V2
V3
V4
V5
V6
1.00
Layer 1
56x56
0.87
0.10
0.01
0.01
0.00
0.00
0.00
0.50
56x56
Layer
0.92
0.03
0.03
0.01
0.00
0.00
0.00
0.00
Layer 3 28x28
-0.25
0.78
0.20
0.00
0.00
0.00
0.00
0.00
-0.50
Layer 4
14x14
-0.75
0.70
0.25
0.02
0.02
0.01
0.01
0.00
1.00
Figure A19: Actual kernels in the spatial domain of a ResNet-18 with additional zero padding before our NIFF trained on ImageNet-100 Still, most kernels exhibit well-localized, small spatial kernels.
23

---

Published in Transactions on Machine Learning Research (05/2024)
Vo
V1
V
V3
V4
V5
V6
1.00
56x56
Layer
0.75
0.51
0.23
0.20
0.02
0.01
0.01
0.01
0.50
Layer 2
56x56
0.25
0.74
0.12
0.06
0.04
0.02
0.01
0.01
0.00
Layer 3
28x28
-0.25
0.65
0.15
0.08
0.04
0.03
0.02
0.01
-0.50
Layer 4
14x14
0.75
0.85
0.10
0.03
0.02
0.00
0.00
0.00
11.00
Figure A20: Actual kernels in the spatial domain of ResNet-18 which mimics lincar convolutions with our NIFF trained on ImageNet-100. We plot for each kernel the zoomed-in (13 x 13) version below for better visibility. Still, most kernels exhibit well-localized, small spatial kernels. However, they are slightly larger than the kernels learned without padding and cropping.
Vo
V1
V2
V3
V4
V5
V6
1.00
Layer
56x56
0.75
0.51
0.23
0.20
0.02
0.01
0.01
0.01
0.50
Layer 2
56x56
0.74
0.12
0.06
0.04
0.02
0.01
0.01
0,00
Layer 3
28x28
0.65
0.15
0.08
0.04
0.03
0.02
0.01
-0.50
Layer 4
14x14
-0.75
0.85
0.10
0.03
0.02
0.00
0.00
0.00
-1.00
Figure A21: Actual kernels in the spatial domain of a ResNet-18 which mimics linear convolutions with our NIFF to mimic linear convolutions trained on ImageNet-100. Still, most kernels exhibit well-localized, small spatial kernels. However, they are slightly larger than the kernels learned without padding and cropping.
24

---

Published in Transactions on Machine Learning Research (05/2024)
Layer 1 56x56
Layer 1 zoomed in 9x9
1.00
Layer 1 56x56
Layer I zoomed in 9x9
Layer 56x56
Layer 2 zoomed in 9xg
Layer 56x56
Layer 2 zoomed 9x9
Layer 3 26x28
Layer 3 zoomed in 9x9
Layer 3 26x28
Layer 3 zoorned in 9x9
Layer 14x14
Layer zoomid in 9x0
zoomed in Layer 4
14x14
Layer
9x9
Figure A22: Actual kernels in the spatial domain of a ConvNeXt--inn including our NIFF trained on
ImageNet-Ik. We plot for each kernel the zoomed-in (9 x 9) version below for better visibility. Overall, most kernels exhibit well-localized, small spatial kernels.
25

---

Published in Transactions on Machine Learning Research (05/2024)
Layer 1 56x56
Layer 1 zoomed in 9x9
1.00
Layer 1 56x56
Layer I zoomed in 9x9
Layer 56x56
Layer 2 zoomed in 9xg
Layer 56x56
Layer zoomed Dvg
Layer 3 26x28
Layer 3 zoomed in 9x9
Layer 3 26x28
Layer 3 zoorned in 9x9
Layer 14x14
Layer xoomed in 9x9
Layer 4 zoomed in Layer 4
14x14
9x9
Figure A23: Actual kernels the spatial domain of a ResNet-50 including our NIFF trained on ImageNet- 1k. We plot for each kernel the zoomed-in (9 x 9) version below for better visibility. Overall, most kernels exhibit rell-localized. small spatial kernels.
26


Reviews:
[{"review_id": "7XlRipQTzT", "reviewer": "Reviewer_wtTg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n* The PCA analysis shown in Fig. 1 is interesting. The observation that large kernel networks still learn well-localized small kernels is useful for network design.\n* The proposed method shows large improvements over ResNet, as shown in Table 2.\n\n---\n### Weakness\n\n* Figure 2 needs to be carefully refined. It is quite difficult to capture the main idea of the proposed method in current form.\n* The novelty and contribution of the proposed method is a bit unclear, since using FFT+IFFT is not a new idea in neural networks (GFNet etc.). More discussions and comparisons should be provided.\n* As shown in Table 2, the proposed method suffers from 2% accuracy drop for DenseNet. \n* The proposed method also does not perform well for mobile networks such as ConvNeXt-Tiny and MobileNet-v2. Besides, more modern mobile networks should be included rather than MobileNet-v2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel approach which enables CNNs to efficiently learn infinitely large convolutional filters. Specifically, this paper introduces MLP parameterized Neural Implicit Frequency Filters (NIFFs) which learn filter representations directly in the frequency domain and can be plugged into any CNN architecture. Experiments are conducted on CIFAR and ImageNet dataset to demonstrate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses part", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pMhl2an3i0", "reviewer": "Reviewer_Mxbe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper proposes a principles way to train CNNs with infinite kernel sizes from the perspective of the frequency domain, which is another success of the math-inspired approach in deep learning. \n\n2. The proposed approach has great potential to reduce the complexity of infinite kernels. The performance demonstrated in Table 2 is quite strong. \n\n3. The analysis of effective kernel size is interesting. \n\n**Weaknesses**\n\nMajor: \n\n1. My biggest concern is the comparisons made in Table 1 and Table 2 are not very fair. In Table 1, the authors compare different frequency-domain approaches using different architectures, which is not very informative to see if the performance gains are obtained by NIFF or the advanced architectures. In Table 2, while using the same architectures, it is not very clear if models are trained exactly using the same configurations or not. After checking the appendix, I realized the models of NIFF are trained with more advanced recipes used by ConvNeXt, which is much stronger than the original one used by ResNet (He et al. 2016). I believe the performance of ResNet in Table 2 is not trained with the same recipes as NIFF, given its relatively low accuracy of 76.1% for ResNet50. Such an unfair comparison again is insufficient to justify the benefits of NIFF. What makes more sense to me is comparing different frequency-domain approaches with exactly the same training recipes and architectures. \n\n2. Besides accuracy, it would be better to report training and inference costs to obtain the final model, as well as the model size including the hypernetworks. \n\n3. Figure 8 only reports the theoretical FLOPs. While I understand it might be out of the scope of this paper for efficient implementation of NIFF, reporting the real training time will give a full picture to the readers. Although in this case, we might not obtain any speedups, I will not see this as a drawback. \n\nMinor:\n\na. figure 2 and figure 3 are not very straightforward to understand the algorithm. I suggest modifying them to improve the presentation of the paper. b. can authors explain why DenseNet-121 NIFF and ConvNeXt-tiny NIFF fall short of the original models significantly? c. some papers appear repeatedly in the reference such as Imagenet: A large-scale hierarchical image database.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a way to build infinitely large convolutions via Neural Implicit Frequency filters. By transferring the learning from the spatial domain to the frequency domain, the proposed approach can reduce the learning complexity from O(N^4) to O(N^2log(N)). Using this technique, they can inpressively scale infinite convolutions to more complex dataset such as ImageNet-1K and ImageNet-100 with very good performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I like the paper a lot. If the major weaknesses can be addressed properly, I will recommend accepting the paper.", "broader_impact_concerns": "No broader impact statement is provided.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f43cU5xTgp", "reviewer": "Reviewer_4rXV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper outlines interesting findings, such as the fact that CNNs do not always seem to benefit from using global conv kernels even if this is possible. It also shows that basically all studied architectures can benefit from a more flexible parameterization than the usual 3x3, 7x7 considered in the baselines.\n\nThe experimental results are very appealing and encouraging for future research.\n\n### Weaknesses\n\nUnfortunately, I am afraid that there are several mixed messages in the paper, both in terms of findings and contributions, which weakens the novelty and validity of the approach as well as its findings.\n\n* Papers that rely on long convolutions, e.g. CKConv, Hyena, S4, S5, just to name a few, all make use of the Fourier theorem to compute convolutions with very large kernels. The paper should make clear that this is part of the main pipeline considered in all existing long conv models. What is new is the fact that the kernels are directly parameterized in the Fourier domain.\n\n* “At the same time, it remains unclear whether there is an optimal filter size and which size of filters would be learned could the model learn arbitrary sizes’ \n+-> As far as I understand, this is exactly the problem that FlexConv is trying to solve. In fact the findings found in that paper actually align very well with the findings obtained here. I know that there are different factors, but it should be clear that this has been done / tackled before.\n\n* “First model to achieve results on par with state-of-the-art on standard high-resolution image classification benchmarks, while executing convs in the frequency domain.”\n+-> \nOn the related work section about Large kernel sizes, several important works are missing, e.g., CKConv, Hyena, S4, S5, etc. There’s an increasing body of literature on long convolutional models.\nIn Section 3, it is assumed that inputs are given in the Fourier domain. But this is clearly not the case, as only convolutions are performed in the Fourier domain. The rest of the operations are spatial. This should be very clear in the paper. \nAn important aspect that ties to the previous point is that the **CCNs trained in the frequency domain** considered here in comparisons try to solve a much more difficult problem. The main objective is to train and deploy CNNs **solely** on the Fourier domain, i.e., to construct networks on which only an FFT is used in the first layer, and an iFFT is used in the last layer of the network. Everything in between happens in the Fourier domain. This task is much more difficult than deploying convolutions on the Fourier domain, as this involves defining nonlinearities, normalization and other layers on the Fourier domain, which is very nontrivial. This difference should also be very clear in methodological and experimental comparisons.\n\n* From what I understand, the only difference between the parameterization of CKConvs and NIFFs conv kernels is the fact that the latter is modeling the kernels directly on the frequency domain. I do not thing this is clear in Sec. 3.1. Also, in 3.1, it is not clear whether $\\Phi$ is defined on complex numbers, or if this is real and only the output is made complex by the concatenation of two channels that correspond to real and imaginary parts. In which part does it become complex-valued? This is important, for instance, in terns of the nonlinearities used. In addition, it says here that the output is C-dimensional. But these values are complex. Depending on how this is parameterized, this could also be 2*C. \n\n* In 3.2, there is a discussion on how Full-Convolutions are implemented. However, it is important to note that this is **not** a full convolution. In fact, what is explained here is the exact definition of depth-wise convolutions. Again, this should be clear and it should also be explained how this impacts the runtime, parameter count and accuracy.\n\n* As explained previously the results in Tab 1 are misleading. AFAIK these architectures try to learn whole pipelines in the Fourier domain.\nIn 2D, long kernels could have rectangular forms, as shown in FlexConv. However, the metric used for the effective kernel evaluation only considers squared shapes. Have the authors considered this? I believe (based on the findings of FlexConv), that this could lead to misleading results. Also, it is clear that downsampling plays a major role on the effective kernel size of a layer. Has this been taken into account? For example, a kernel of size 14 in the 4th layer of ConvNext-tiny (Fig 4) is actually pretty much global in terms of the original resolution. This should be very clear in the paper to avoid misleading conclusions.  \n\n### Questions / Comments\n\n* The paper makes use of circular convolutions on the Fourier domain, but it is also possible to define proper (non-circular) convolutions by zero padding the input. Is there a clear understanding of how this difference affects the results? For comparison, it could be possible to approximate this setting by training CNNs that use cyclic padding instead of zero padding. Or by padding feature maps pre FFT to avoid this.\n\n* In FlexConv, the authors explicitly mention limitations related to the use of FFTConv. However, from what I understand, you did not experience these limitations. Could you comment on that?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a study on the size of the optimal kernel size of different neural architectures. To this end, it proposes to parameterize conv kernels directly in the Fourier domain as an implicit neural representation, such that kernels of different sizes can be parameterized under the same number of parameters.\n\nThe study proposed in this paper is very interesting, as it concerns the family of long convolutional models, which have been becoming more popular in recent years.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe that the study presented in this paper is important and relevant to the TMLR community. However, there are several points that convolute the message, the veracity and the conclusions of the paper. I believe that the paper requires a lot of rewriting and complementation before acceptance, which prevent me from directly recommending acceptance.\n\nWith that being said, I am open to support acceptance of the paper, should the authors address these concerns posed in the previous section during the discussion period.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AE sides with acceptance in accordance with the thorough and detailed reviews on the technical content, empirical results, and overall claims and evidence in the submission. While one official recommendation leaned toward rejection, there is nevertheless consensus among two of the three reviewers and the AE. The argument for rejection is a complaint that fourier domain processing already exists in deep learning ("FFT+IFFT is not a new idea in neural networks (GFNet etc.)") and that certain convolutional architectures do not perform well with the method ("does not perform well for mobile networks such as ConvNeXt-Tiny and MobileNet-v2"), as well as a drop in accuracy for DenseNet. This argument does not engage with the full scope of the submission, or its chosen claims about exploring and applying adaptive and large kernel sizes, which are well-connected to the evidence.

The featured certification is recommended by the AE to (1) second the recommendation for this certification by a reviewer and (2) to reflect the satisfaction—enthusiasm even—by two of the three reviewers for the topic and execution of this work. It remains important to study fundamental factors of deep learning for vision, such as the degree of locality / the size of kernels, along with the pursuit of ever better general metrics like accuracy and efficiency.

The level of detail and background for each review offers more context for the decision. All three reviewers are experts on deep learning and vision with particular expertise in convolutional and attentional architectures, specialized filtering, and searching or optimizing over architectures/parameterizations for accuracy and generality. The reviewers Mxbe and 4rXV are the most positive and have the most expertise on filtering, the design and optimization of architectures for different scopes/resolutions/sparsity,  and implicit modeling. To be clear, the review and perspective of wtTg focusing on convnet architectures and baselines is also informative, and the AE likewise appreciates their service. The decision depends on the depth and breadth of each review, and the AE's own background on learning and filtering, and so to summarize the recommendations were weighted by their level of detail and correspondence with the full content of the submission.

The AE thanks the authors and reviewers for engaging in the TMLR process to deliver an improved paper with agreement between its claims and evidence and a clear audience.
Decision: Accept (poster)