Why Convolutional Networks Learn Oriented Bandpass Filters: Theory and Empirical Support
Authors: Isma Hadji
Richard Wildes
Venue:  (2021)

Abstract: It has been repeatedly observed that convolutional architectures when applied to image understanding tasks learn oriented bandpass filters. A standard explanation of this result is that these filters reflect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass filters are matched to such structure. We offer an alternative explanation based not on the structure of images, but rather on the structure of convolutional architectures. In particular, complex exponentials are the eigenfunctions of convolution. These eigenfunctions are defined globally; however, convolutional architectures operate locally.  To enforce locality, one can apply a windowing function to the eigenfunctions, which leads to oriented bandpass filters as the natural operators to be learned with convolutional architectures. From a representational point of view, these filters allow for a local systematic way to characterize and operate on an image or other signal. We offer empirical support for the hypothesis that convolutional networks learn such filters at all of their convolutional layers. While previous research has shown evidence of filters having oriented bandpass characteristics at early layers, ours appears to be the first study to document the predominance of such filter characteristics at all layers. Previous studies have missed this observation because they have concentrated on the cumulative compositional effects of filtering across layers, while we examine the filter characteristics that are present at each layer.

Under review as a conference paper at ICLR 2021
WHY CONVOLUTIONAL NETWORKS LEARN ORI- ENTED BANDPASS FILTERS: THEORY AND EMPIRICAL SUPPORT
Anonymous authors Paper under double-bling brindd review
ABSTRACT
It has been repeatedly observed that convolutional architectures when applied image understanding tasks learn oriented bandpass filters. A standard explanation of this result is that these filters reflect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass filters are matched to such structure. We offer an alternative explanation based not on the structure of images, but rather on the structure of convolutional architectures. particular, complex exponentials are the eigenfunctions of convolution. These eigenfunc- tions are defined globally; however, convolutional architectures operate locally. To enforce locality, one apply windowing function to the eigenfunctions, which leads to oriented bandpass filters as the natural operators to be Iearned with convolutional architectures. From representational point of view, these filters allow for a local systematic way to characterize and operate on an image or other signal. We offer empirical support for the hypothesis that convolutional networks learn such filters at all of their convolutional layers. While previous research has shown evidence of filters having oriented bandpass characteristies at early layers, ours appears to be the first study to document the predominance of such filter characteristies all layers. Previous studies have missed this observation because they have concentrated on the cumulative compositional effects of filtering across layers, while we examine the filter characteristies that are present at each layer.
INTRODUCTION
1.1 MOTIVATION
Convolutional networks (ConvNets) conjunction with deep learning have shown state-of-the-art performance in application computer vision, ranging across both classification (e.g.. Krizhevsky
ct al. (2012): Tran et al.) (2015); Ge et al. (2019)) and regression ec.g..(zegeedy et al. (2013) Eigen & Fergus (2015): ) Zhou al. (217) ) tasks. However, understanding of how these systems achieve their remarkable results lags behind their performance. This state of affairs is unsatisfying not only from a scientific point of view, but also from an applications point of view. As these systems move beyond the lab into real-world applications better theoretical understanding can help establish performance bounds and increase confidence in deployment.
Visualization studies of filters that have been learned during training have been one of the key tools marshalled lend insight into the internal representations maintained by ConvNets in application to computer vision, e.g.. Zeiler & Fergus (2014); Yosinski et al. (2015): Mahenndan & Vedaldi (2015); Shang et al. (2016): Feichtenhofer et al. (2018). Here, an interesting repeated observation that early layers in the studied networks tend to learn oriented bandpass filters, both in two image spatial dimenstions, (ar, y) in application to single image analysis as well as in three spatiotemporal dimensions, (z, y, t) in application to video. An example is shown in Fig. 1. Emergence of such filters seems reasonable, because local orientation captures the first-order correlation structure of the data, which provides reasonable building block for inferring more complex structure (e.g., local measurements of oriented structure can be assembled into intersections to capture corner structure, etc.). Notably, however, more rigorous analyses of exactly why oriented bandpass filters might be

---

Under review as a conference paper at ICLR 2021
Figure 1: Visualization of pointspread functions (convolutional kernels) previously observed to be learned in the early layers ConvNets. Brightness corresponds to pointwise function values. The majority of the plots show characteristics of oriented bandpass filters in two spatial dimensions, i.e., oscillating values along one direction, while remaining relatively constant in the orthogonal direction, even as there is an overall amplitude fall-off with distance from the center. The specific examples derive from the early layers of ResNet-50 architecture He et al. (2016) trained on Ima- geNet Russakovsky et al. 2015)).
learned has been limited. This state of affairs motivates the current paper in its argument that the analytic structure of ConvNets constrains them to learn oriented bandpass filters.
1.2 RELATED RESEARCH
Visualization of receptive field profies (i.e., pointspread functions Lim (9900) of the convolutional filters learned by contemporary ConvNets is a popular tool for providing insight into the image properties that are being represented by network. A notable trend across these studies is that early layers appear to learn oriented bandpass filters in both two spatial dimensions, e.g., Zeiler & Fergus (2014); Springenberg et al. (2015); Yosinski al. (2015); Shang et al. (216)), well three spa- tiotemporal dimensions, e.g., Feichtenhofer et al. (2018). Indeed, earlier studies with architectures that also constrained their filters to be convolutional in nature, albeit using Hebbian learning strat- egy MacKay/ (2003) rather than the currently dominant proppaaoonntioon approach Rumelhart et al. (1866). also yielded filters that visualized having oriented bandpass filter characteristies Linsker 1986.. Interestingly, biological vision systems also are known to show the presence of oriented bandpass filters their earlier layers of processing in visual cortex; see Hubel Wiesel (1922) for pioneering work along these lines and for more general review De Valois & De Valois 1988).
The presence of oriented bandpass filters in biological systems often has been attributed their be- ing well matched to the statistics of natural images Field (1987): Olshausen & Field (1996); (arklli Lewicki (2009); [Simoncelli & Olshausen (2001), e.g., the dominance of oriented contours mul- tiple scales. Similar arguments have been made regarding why such filters are learned by ConvNets. Significantly, however, studies have shown that even when trained with images comprised random noise patterns, convolutional architectures still learn oriented bandpass filters Linsker (1866). These latter results suggest that the emergence of such filter tumings cannot be solely attributed to systems being driven to learn filters that were matched to their training data. Similarly, recent work showed that randomly initialized networks serve well in image restoration problems Ulyanov et al. (018))
2000):|mmmonelll
Some recent multilayer convolutional architectures have specified their earliest layers have ori- ented bandpass characteristics, e.g.. Bruna & Mallat (2013); Jacobsen et al. (2016; Hadji & Wildes (017)); indeed, some have specified such filters across all layers [Brnna Mallat (2013): Hadjj & Wildes (2017). These design decisions have been variously motivated in terms of being well matched to primitive image structure Hadji & Wildes (2017) or providing useful building blocks for learning higher-order structures Jacobsen et al. (2016) and capturing invariances Bruna & Mal- lat (2013). Other work has noted that purely mathematical considerations show that ConvNets are well suited to realizing filter designs for capturing multiscale, windowed spectra Bruna et al. (2016); however, it did not explicitly established the relationship to eigenfunctions of convolution nor offer an explanation for why deep-learning yields oriented bandpass filters when applied to ConvNets. It also did not provide empirical investigation of exactly what filter characteristics are learned each convolutional layer of ConvNets.
2

---

Under review as a conference paper at ICLR 2021
1.3 CONTRIBUTIONS
In the light of previous research, the present work appears to be the first to offer an explanation of why ConvNets learn oriented bandpass filters, independently of the input, by appeal to the inherent properties of their architectures in terms of their eigenfunctions. By definition, the convolutional layers of a ConvNet are governed by the properties of convolution. For present purposes, a key property is that the cigenfunctions of convolution are complex exponentials. Imposing locality on the eigenfunctions leads oriented bandpass filters, which therefore are the appropriate filters to be learned by ConvNet. Indeed, these theoretical considerations suggest that oriented bandpass filters should be learned at all layers for this observation by examining filters across all convolutional layers of three standard ConvNets (AlexNet Krizhevsky et al. (0122), ResNet He et al. (2016) and VGG16 Simonyan & Zisserman (014)) and show that both numerically and visually they are well characterized as having learned oriented bandpass filters at all their convolutional layers. Our empirical study is distinct from earlier visualization efforts, which concentrate the cumulative compositional results of filtering across layers that typically show emergence of complicated structures in the layerwise feature maps, while focus on the complementary question of what primitive filter characteristics have been learned at each individual layer and offer both numerical as well as visualization analyses.
ConvNet, not just at early layers. We provide empirical support
2
THEORY
This section details a novel explanation for why ConvNets learn oriented bandpass filters. The first two subsections largely review standard material regarding linear systems theory Oppenheim et al. (1983) and related topics Kaiser (20TT); Kusse & Westwig (2006), but are necessary to motivate properly our explanation. The final subsection places the material in the context of ConvNets.
2.1 EIGENFUNCTIONS OF CONVOLUTION
Let L a linear operator on a function space, The set of eigenfunctions on associated with this operator satisfy the condition Kusse & Westwig (2006)
LΦn An
That is, the operator acts on the eigenfunctions simply via multiplication with a constant, An. re- ferred as the eigenvalue. It sometimes also is useful to introduce a (positive definite) weighting function, w, which leads to the corresponding constraint
Cø,, X, WoO..
For cases where any function in the space can be expanded as a linear sum of the eigenfunctions, is said that the collection of cigenfunctions form complete set. Such set provides convenient and canonical spanning representation.
Let (I1, ( T2..... ..., TT,T_ (a1, ag...... ..., TT) and = (w1, U2.... T,, For the space of convolutions, with the convolution of two functions, fxx and h(x) defined as
/
f(x) h(x)=
f(x- a)h(a) da
(3)
it is well known that functions of the form f(x) = Giu x are eigenfunctions of convolution Oppen- heim et al. (1983), i.e.,
in T (x (x-a) da = ciu foo e e-in" -innTaaaa a da
(4)
with the equality achieved via appealing to ein" (x-n) din xx-xx" Tn and subsequently factoring eiu" x outside the integral as it independent of a. The integral on the right hand side of (4
"h(a) da,
(5)
the eigenvalue, referred as the modulation transfer function (MTF) in signal processing Oppen- heim et al. (183). Noting that ei x = cos(u" Tx) + sinn(u x) leads to the standard interpretation of in terms of frequency of the function (e.g., signal).
3

---

Under review as a conference paper at ICLR 2021
Given the eigenfunctions of convolution are parameterized in terms of their frequencies, it is useful to appeal to the Fourier transform of function f(x), where we use the form Hom 1986)
F(u) / f(x)e iu" dxc
because any convolution can be represented in terms of how it operates via simple multiplication of the eigenvalues, (5). with the eigenfunctions, e x, with given by (6). Thus, this decomposition provides a canonical way to decompose f(x) and explicate how a convolution operates on it.
2.2 IMPOSING LOCALITY
Understanding convolution purely in terms its eigenfunctions and eigenvalues provides only a lost in the global transforma-
global representation of operations, as notions of signal locality, tion to the frequency domain, u. This state of affairs often is unsatisfactory from a representational understand the structure of the signal (e.g.. image) on more
point of view because local basis (e.g., one wants to detect objects as well as their image coordinates). This limitation can be ameliorated by defining a windowed Fourier transform Kaiser (20IT), as follows Jahne & Hausbecker (2000).
Let (x) be windowing function that positive valued, symmetric and monotonically decreasing from its center so as to provide greatest emphasis at its center. A Windowed Fourier Transform (WFT) of (x) can then be defined as
[[ { f(a)w((- x)e- -u da.
Making use of the symmetry constraint that we have enforced on the windowing function allows for wxx) w(-x) so that the WFT, 7 be rewritten as
F(u: w) f(a)w(x a)e (x- _ da,
(8)
which has the form of convolution
f(x) (wx))euxxx)
with the inclusion of an additional phase component, e -iu
To provide additional insight into the impact the WFT convolution, has on the function, fxx). it is useful to examine the pointspread function, wxx)e""! in the frequency domain by taking its Fourier transform (6). i.e., calculate its MTF. We have
w (x)e x
xi-=" -iu dx,
(10)
which via grouping by coefficients of becomes
wxx)e u() u.) dx.
(11)
Examination of (TT) reveals that it is exactly the Fourier transform of the window function, cf. (6). shifted to the center frequencies, u. Thus, operation of the WFT convolution, (9). on function, fxx), passes central frequency, u, relatively unattenuated, while suppresses those that further away from the central frequency according to the shape of the window function, w(x), ie., it operates as a bandpass filter. Thus, convolution with bank of such filters with varying central frequencies, u,, has exactly the desired result of providing localized measures of the frequency content of the function fx)).
Returning to the pointspread function itself, w(x)e"", x and recalling that e = cos(u x) + é sin(u, u. x). it is seen that in the signal domain, x, the filter will oscillate along the direction of u, while remaining relatively constant in the orthogonal direction, even as there is overall amplitude fall-off with distance from the center according to the shape of w(x), i.c., we have an oriented bandpass filter.

---

Under review as a conference paper at ICLR 2021
Figure 2: Visualization of an analytically defined oriented bandpass filter (12) The left panel shows the pointspread function corresponding to the odd symmetry (sin) component, while the right panel shows its power in the frequency domain. Brightness corresponds to pointwise function values.
As a specific example Jahne Haushecker 2000). taking w((x) to be an n-dimensional Gaussian- like function, g(x o) = K - -xxllx////"", with a the standard deviation and x. scaling factor, yields an n-dimensional Gabor-like filter,
o(x;o)e""" g(x: (r) (cos(u,) (u. x) + issn(u x))
(12)
which provides good joint localization of signal content in the signal and frequeney domains Gabor (1946). Indeed, visualization of these filters in two spatial dimensions (Figure 2) provides strik- ingly similar appearance to those presented in Figure 1, if in an idcalized form. In particular, their pointspread functions oscillate according to frequency ||u,]| along the direction, [[ while maining relatively constant in the orthogonal direction, even as there an overall amplitude fall-off with distance from the center. In the frequency domain, they have peak power at u. with fall-off following a Gaussian-lik like shape with standard deviation, 1/0r, that is the inverse of that used in specifying the window, w(x). These observations hold because we already have seen, (11), that the frequency domain representation of such a function is the Fourier transform of the window function, wxx), shifted to the center frequencies, u,: furthermore, the Fourier transform of function of the form (x; o) has a similar form, albeit with an inverse standard deviation Bracewell 1986).
2.3 IMPLICATIONS FOR CONVNETS
Convolutions in ConvNets serve to filter the input signal to highlight its features according to the learned pointspread functions (convolutional kernels). Thus, convolution with the oriented filters shown in Figure 1 will serve to highlight aspects of an image that are correspondingly oriented and at corresponding scales. The question hand is, "Why did the ConvNet learn such filters?" The previous parts of this section have reviewed the fact that complex exponentials of the form ciu"x_ " x) + sin(n"" x) are the eigenfunctions of convolution. Thus, such frequency de- pendent functions similarly serve as the eigenfunctions of the convolutional operations in ConvNets. In particular, this result basic property of the convolutional nature of the architecture, indepen- dent of the input to the system. Thus, for any convolution in a ConvNet the frequency dependent eigenfunctions, e provide systematic way to represent their input.
As with the general discussion of locality presented in Subsection 2.2 for the specifics of ConvNets it also is of interest be able characterize and operate locally signal. At the level of convolu- tion, such processing is realized via pointspread functions that operate as bandpass filters, (9). Like any practical system, ConvNets will not capture continuous range of bandpass characteristics, as given by u, and the sampling will be limited by the number of filters the designer allows at each layer, i.e., as a metaparameter of the system. Nevertheless, making use of these filters provides systematic approach to representing the input signal.
Overall, the very convolutional nature of ConvNets inherently constrains and even defines the filters that they learn, independent of their input or training. In particular, learning bandpass filters provides a canonical way to represent and operate on their input, as these serve as the localized eigenfunc- tions of convolution. As ConvNet is exposed to more and more training data, its representation is optimized by spanning as much of the data as it can. Within the realm of convolution, in which ConvNet conv layers are defined, oriented bandpass filters provide the solution. They arise as the lo- cality constrained eigenfunctions of convolution and thereby have potential to provide a span of any input signal in localized manner. Thus, ConvNets optimized by learning exactly such filters. Notably, since this explanation for why ConvNets learn oriented bandpass filters is independent of training data, it can explain why such filters emerge even when the training data lacks such pattern
5

---

Under review as a conference paper at ICLR 2021
structure, including training on random input signals, e.g., Linsker (1986). Moreover, the explana- tion is independent of the learning algorithm as any algorithm driving its learned representation to span the space of input signals achieves its goal in the cigenfunctions of the convolutional architec- ture, i.e., oriented bandpass filters. Sec. 1.2 reviewed work showing that both back propagation and Hebbian learning yield oriented bandpass filters as their learned convolutional representations.
Our analysis of oriented bandpass filters as the localized eigenfunctions of convolution is not specific early ConvNet layers, but rather applies to any convolutional layer. The result thereby makes a theory-based prediction that oriented bandpass filters should be learned at all conv layers in a ConvNet. As reviewed in Sec. [.2] previous studies have demonstrated that filters learned at early ConvNet layers visualize as oriented bandpass filters; however. it appears that little attention in previous studies has examined the pointspread functions of learned convolutional filters deeper in network. Instead, studies of learned filters deep in a ConvNet have focused on the cumulative effect of filtering across all layers upto and including a particular layer under consideration. In the following, we present complementary study that empirically examines filters that have been learned at a given layer without reference to previous layers to see whether they appear as oriented bandpass filters.
EMPIRICAL SUPPORT
In this section, we present empirical support for the theory- prediction that oriented bandpass filters should be learned all layers in a ConvNet. We examine three standard ConvNets Net Krizhevsky et al. (2012), ResNet50 He et al. (2016) and VGG16 (Smoonyn & Zisserman (2014)) from both numerical and visualization perspectives. In all cases, we make use of publicly available implementations of the architectures AlexNet ResNet VGGI6
3.1 NUMERICAL. STUDIES
To study numerically whether ConvNets learn oriented bandpass filters, we perform least-squares fit of the derived oriented bandpass filter, (12), to all learned convolutional filters at all layers of each model. While other oriented bandpass filter models can be used here, the model considered, (12). is a natural choice in the present context, as it results from our theoretical analysis in Sec. 2 In particular, we fit the free parameters of a 2D instantiation of the model, (i.e., the center frequencies, ue, and the standard deviation, r), to the learned pointspread values of every convolutional filter at every layer of (eeNee//Krizhvssee et al.] (012), ResNee50 He al. (2016) and VGG16 [Sionnyan & Zisserman (0144). Finally, we take the root-mean-square (RMS) error residual between each individual fit of the model and the corresponding learned pointspread function as indicative of how well the learned filter is captured by oriented bandpass filter, with smaller error indicative of better fit. Results are plotted in Fig. 3.
For all three architectures the histograms of residuals collapsed across layers shows that in all cases the fitting errors mostly lie below 0.1, with the bulk of errors residing below 0.01 and lower. These small residuals indicate generally good fits of the learned filters to the oriented bandpass model.
A more detailed look can be had by considering box plots of errors by layer. Here, the results for AlexNet, for example, show that the median fitting error is under 0.04 at layer one and subsequently decreased to under 0.02 for all subsequent layers as well as for the aggregated fit across all layers. Moreover, 95% of the data lies under 0.15 at layer 1 and under 0.04 all other layers as well as the aggregate across layers. To place these numbers in perspective, we perturbed the otherwise analytically defined pointspread function, (2)), with various amounts of noise and then compared to the same function without corruption to see how much corruption would yield various RMS errors. Results are summarized in Fig. 4. For example, we find that random noise within only ~6% of the distribution of the uncorrupted filter values yields 0.04 residual, which demonstrates that the discrepancy from the purely analytic form is very small, indicating that the observed fits quite good. Still, indicated by the raw histograms, Fig. 3, some outliers observed, where the errors approach 0.1 and beyond. Visual inspection of these cases indicates that they arise when the learning process apparently has failed, the learned filter has essentially constant valued pointspread function (i.e., is flat) or else has no discernible structure.
Results for ResNet50 and VGG16 show similar patterns to those of AlexNet, with the main dif- ference being that the distributions are shifted to larger values at their first layers; however, they subsequently conform to values similar to those of Alex thereafter. Also, it is seen that the layer

---

Under review as a conference paper at ICLR 2021
Alvollet
20000
20000
0.00
0.00
18-3
0.37
0.10
60000
60000
0.00
49000
30000
0:00
20000
30000
0.023
33000
9966
Figure 3: RMS errors from fitting an oriented bandpass model convolutional filters learned by AlexNet Krizhevsky et al. (012) (top row), ResNet50 He et al. (2016) (middle row) and VGGI6 Simonyan & Zisserman (2014) (bottom row). Left column shows histograms of residuals collapsed across all layers for each architecture: Residual values and count along the abscissa and ordinate, resp. Right column shows box plots Tukey (1977) of residuals by layer along abscissa, with the last plot for each architecture showing results across all layers: The box in each column encompasses the observed interquartile range, the horizontal line in each box indicates the median value, the whiskers above and below the box extend to the 95th and 5th percentiles, resp.
0.339
A18
0070
0000
>.comagetie
Figure 4: RMS error versus amount of corruption added to an analytically defined oriented bandpass filter yield that eиror compared the uncorrupted filter.
one distribution for VGG16 is shifted upward relative to ResNet50. Perhaps this pattern arises be- cause the layer one filter sizes are largest for AlexNet and smallest for VGG16, with ResNet50 lying in between, while at layer two and beyond VGG16 and ResNet50 have the same size filters and all three architectures have the same size filters at layer four and beyond. Interestingly, for all archi- tectures it is seen that there is a marked decrease in the residual values between layers one and two. This result parallels observations made in earlier visualization studies that learned filters typically begin to show more strongly oriented structure after the first layer, e.g., Linsker (1986); Springen- berg et al. (2015). Overall, these numerical experiments indicate that all three of the considered architectures learn oriented bandpass filters at all layers.

---

Under review as a conference paper at ICLR 2021
Layer
Layer
Layer 3
Layer
Layer
Layer
Layer 7
Layer 8
Figure 5: Visualization of learned pointspread functions for AlexNet Krizhevsky et al. (2012). Shown are visualizations of representative functions that have median residual values at each layer. For each layer, the plot the left shows the leamnd function and the plot the right shows the correspondingly fit function. Brightness corresponds to pointwise function values.
3.2 VISUALIZATION STUDIES
For visualization studies, we plot the pointspread functions for selection of the learned filters each layer and display them as images. the interest space, here we focus on AlexNet; although, the visualization results for the other architectures similarly bandpass filters, as would expected from the numerical results of Sec. [.1 particular, Fig. shows plots of learned pointspread functions from all layers of AlexNet. The shown pointspread functions are representative
supportive of oriented
the median residual value for each layer, and they are paired with visualization of the corresponding fit to the oriented bandpass model, (12). Inspection of these plots show that in all cases oriented structure is visible in the learned pointspread functions: In particular, orientations
proceeding from layers right to lower left, vertical, diagonal upper right to lower left, vertical, vertical, diagonal upper left lower right, horizontal and vertical, resp. Moreover, the learned and fit pointspread functions are qualitatively very similar. Moreover, the visualizations suggest improved fits between the learned and fit models layer increases, similar to what is seen the numerical results of Sec. 331 Overall, these visualization results corroborate the numerical results indicating that oriented bandpass filters are indeed learned at layers of the considered ConvNets.
manifest approximately along slight diagonal upper
SUMMARY
Previous studies have demonstrated that learned filters the early layers of convolutional networks visualize as oriented bandpass filters. This phenomenon typically is explained via appeal to natural image statistics, i.e., natural images are dominated by oriented contours manifest across variety of scales and oriented bandpass filters are well matched to such structure. We have offered an alter- native explanation in terms of the structure convolutional networks themselves. Given that their convolutional layers necessarily operate within the space of convolutions, learning oriented band- pass filters provides the system with the potential to span possible input, even while preserving a work is applicable not just early ConvNet
notion of locality in the signal domain. Notably, layers, but to all conv layers in such networks. We have provided empirical support for this claim, showing that oriented bandpass filters indeed learned all layers of three standard ConvNets. These results not only provide new insights into the operations and representations learned by Con- vNets, but also suggest interesting future research. In particular, our work motivates investigation of novel architectures that explicitly constrain their convolutional filters to be oriented bandpass in learning-basse based framework. such framework, would not be necessary for the training process to learn the numerical values for each and every individual filter value (i.e., each filter tap), but rather would merely need to learn a much smaller number of parameters, e.g., the values of the center fre- quency, ue and the standard deviation, ø, associated with the Gabor-like filter derived above, [12 or some other suitably parameterized filter. Such a constrained learning approach would require much less intensive training procedure (e.g. involving far less data) compared to learning values for all individual filter taps, owing to the drastically reduced number of parameters that need to be estimated, while being able tune to the specifies of the task that is being optimized.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
AlexNet. alexnet
R. Bracewell. The Fourier Transform and Its Applications. McGraw-Hill, NY, NY, 1986.
J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell., 35:1872-1886, 2013.
J. Bruna, S. Chintala, Y. LeCun, S. Piantino, A. Szlam, and M. Tygert. A mathematical motivation for complex-valued convolutional networks. Neural Computation, 28:815-825, 225,, 2016.
R. Valol and K. DeValois. Spatial Vision. Oxford University Press, NY, NY, 1988.
D. Eigen and Fergus. Predicting depth, surface normals and sematic labels with a common muli- scale convolutional architecture. In ICCV, 2015.
C. Feichtenhofer, A. Pinz, R. Wildes, and A. Zisserman. What have we learned from deep represen- tations for action recognition? In CVPR, 2018.
D. Field. Relations between the statistics of natural images and the response properties of cortical cells. Journal of the Optical Society of America A, 4:2379, 1987.
D. Gabor. Theory of communication. Journal of the Institute of Electrical Engineers, 93:429-457, 1946.
W. Ge, X. Lin, and Y. Yu. Weakly supervised complementary parts models for fine- fine-grained image classification from the bottom up. CVPR, 2019.
I. Hadji and R. Wildes. A spatiotemporal oriented energy network for dynamic texture recognition. In ICCV, 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
B. Horn Robot Vision. MIT Press, Cambridge, MA, 1986.
D. H. Hubel and T. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 160:106-1544 154, 1962.
J H. Jacobsen, V. Gemert, Z. Lou, and A W.M. Smeulders. Structured receptive fields cnns. In CVPR, 2016.
B. Jahne and Hausbecker. Computer Vision and Applications. Academic Press, San Diego, CA, 2000.
G. Kaiser. Friendly Guide to Wavelets. Modern Birkhauser Classics, Switzerland, 2011.
Y. Karklin and M. Lewicki. Emergence of complex cell properties by learning to generalize in natural scenes. Nature, 457:83-866 2009.
A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
B. Kusse and W. Westwig. Mathematical Physics. Wiley- VCH, Weinheim, FRG, 2006.
J. Lim. Two-Dimensional Signal and Image Processing. Prentice Hall, Upper Saddle River, NJ, 1990.
R. Linsker. From basic network principles to neural architecture. Proc. National Academy of Sci- ences USA, 03:7508-7512. 8390-8394, 8779,8783, 1986.
D. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, Cambridge, UK, 2003.

---

Under review as a conference paper at ICLR 2021
A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.
B. Olshausen and D. Field. Emergence of simple-cell field properties by learning a sparse code for natural images. Nature, 381:607-609, 1996.
A. Oppenheim, Willsky, and I. Young. Signals and Systems. Prentice Hall, Upper Saddle River, NI, 1983.
ResNet. v01/v0.1
D. Rumelhart, G. Hinton, and R. Williams. Learning representations by back-propagating error. Nature, 323:533-536, 1986.
O. Russakovsky, Deng, H. Su, Krause, Satheesh, S. Ma, Huang, Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. IJC,, 2015.
W. Shang, K. Sohn, and Lee D. A. Enlitic. Understanding and improving convolutional neural networks via concatenated rectified linear units. In ICML, 2016.
E. Simoncelli and B. Olshausen. Natural image statisties and neural representation. Annual Review of Neuroscience, 24:1 193-1216, 2001.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2014.
Springenberg. A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convo- lutional net. In ICLR Workshop, 2015.
C. Szegedy, A. Toshev, and D. Erhan. Deep neural networks for object detection. In NIPS, 2013.
D. Tran, L. Bourdev, Fergus, Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
J. Tukey. Exploratory Data Analysis. Addison-Weste Wesley, 1977.
D. Ulyanov, A. Vedaldi, and V. Lempissky, Deep image prior. In CVPR, 2018.
VGG16. http:///ttthccoooc/cce////ee/eee 10q/v0.1
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. In ICML workshops, 2015.
M. Zeiler and Fergus. Visualizing and understanding convolutional networks. ECCV, 2014.
Zhou, M. Brown, N. Snavely, and D. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
10


Reviews:
[{"review_id": "yXV8kyo7UNF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an explanation of why convolutional neural networks learn oriented bandpass filters - as has been commonly shown for early layers in various ConvNet architectures. The main argument is that oriented bandpass filters are the eigen-functions of localized convolution operators and in order to span the input signal space (regardless of its a structure) the network will always learn these functions as filters. Additionally, because this result is independent of input signal it should happen at all layers. These are demonstrated by examining filter of several trained neural network architectures and fitting them to Gabors - showing the fit is good with low residual error.\n\nAll in all I think this is a very interesting paper which addresses a fundamental question and empirical result observed many times in the past. There are, however, many open questions here, many of them I think the paper should have addressed:\n\n* rotation of the basis functions - the main argument of the paper is that these filters are learned because the are the eigen-functions of the convolutional operator and any task that requires spanning the input space (==all tasks) will learn these. However, there are many other possible basis function combinations that would still span the input space but will not be eigen-functions i.e diagonlized. For example, any rotation of these learned filters would be such a basis (but the filters would not be oriented band-pass filters in that case). I don't think there is a good explanation here as to why this specific basis is learned out of all possible bases.\n\n* loss, learning etc - although the authors mention these results are independent of learning algorithm or loss, I would have wanted to see this demonstrated. Either by choosing a different task, a different optimisation method or some combination of the two. Specifically, demonstrating these results hold on non image data would have been more convincing I think (see below).\n\n* discretization - all the analysis in the paper is for the continuous signals, operators and convolutions. Do results change for the discrete case (which is the case here)? If so how? this is not discussed in the paper.\n\n* filter support and size - related to the previous point - how do results change with different filter sizes? the filters used in some of these networks are only 3x3 - this may result in a very coarse estimate for the bandpass-oriented filter parameters. I would like to see some discussion of this.\n\n* no convolutional model also learn such filters - older, non convolutional models also learn oriented band pass filters (sparse coding, ICA etc.) when trained on natural images. These results, as the authors suggest, are related to the structure of natural images - is this related in any way to the proposed explanation? I think this requires more discussion - especially since the statistics of natural images is what makes convolutions such a natural choice for processing them. \n\n* why do later layers get even better?  - it seems that deeper layers, across all architectures, get lower residual errors. Why is this? does it have to do with filter size? any other explanation?\n\n* statistics of orientations and scales - is there any different in orientation and scale statistics for different layers and architectures? this should be easy to obtain and visualize and I think would add to the breadth of results.\n\nBottom line - I think this is an interesting paper which can be even better with more in-depth discussion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lznvMxmNww2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The continuous domain formalism is not adapted to what the authors would like to prove because convnet discrete kernels make inadequate and questionable, any inference through a continuous formalism: one can say anything we want, giving that one can choose the intrinsic underlying continuous kernel as we want. \nLet us develop:\nIn convnets, any convolution kernel h is associated with a discrete (countable) sequence of coefficients and this, necessarily (one cannot store an arbitrary infinite sequence on a computer). Coefficients of h can be seen as samples of an underlying continuous domain function g.\nThe filter corresponding to the discrete sequence h is H (Fourier transform of the impulse response h) and the filter corresponding to g is G. H is periodic even if G is not periodic. So, we can forget the other periods and focus only on a single period. Thanks to that, H is necessarily bandlimited (no need of explanation). For a pure “bandlimited property”, one needs to consider an infinite length sequence h. I hope the authors can prove the contrary because otherwise, there is no need for more explanation.\nI also found a paper [1] that shows that filters learned are restricted to more or less effective “band-limitation” mainly depending on the length of h (3x3 for VGG or up to 11x11 for alexnet). The conclusion is that if you use convolution kernels with short size (2x2 or 3x3 for instance), so only very few filters can have nearly zero magnitude in their Fourier transform and the bandlimited assumption is not really a sharp forcing to zero, but a weighted frequency penalization. This tends to prove my remark above. Maybe it is more convenient to use different terminologies than those used in standard signal processing based on Shannon sampling theory (from continuous to discrete, in contrast with the intrinsic discrete definition of standard convnets): remember that there is no unique way for making inference from discrete samples towards a continuous-time / continuous-spatial process. Bandlimited seems to make no sense when using a purely discrete-time formalism.\nMaybe the most important contribution of the paper could have been explaining the main orientation properties of the selective parts of the convolution filters. In that case, a proof of explanation is more straightforward when generating oriented training datasets and proving that the filters learned are oriented in the same directions.\n[1] A. M. Atto, R. R. Bisset and E. Trouvé, \"Frames Learned by Prime Convolution Layers in a Deep Learning Framework,\" in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2020.3009059.\nI insist (update): band-limited is a very very inappropriate terminology for convnets. I am waiting for author's' answers (need for clarifications): the paper needs to be revised.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "jq3ccfFx8k7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper gives a signal processing analysis of why CNNs learn oriented bandpass filters and provide empirical evidence by analyzing AlexNet, VGGNet, and ResNet trained on ImageNet.\n\nStrengths:\n1. The problem is indeed worth investigating. This type of analysis may have a profound impact on understanding (at least) many low-level (e.g. denoising, super-resolution) and middle level (e.g., boundary detection, texture analysis) algorithms based on CNNs.\n2. The authors give very nice references and credits to previous related work, which is not commonly seen when the whole community enters the era of deep learning.\n3. The signal processing analysis is a nice try.\n\nWeaknesses:\n1. The reviewer is not convinced by the analysis provided in the paper. The reviewer argues that the types of filters learned by CNNs should be task-dependent. For example, for image denoising, the CNNs should be oriented to (effective) low-pass filters; while for edge detection, the CNNs may be encouraged to learn high-pass filters. As for ImageNet (or generic) classification, the CNNs tend to learn oriented bandpass filters.\n2. The reviewer agrees that $w(x)$ should be localized, but it is not necessarily lowpass (as  Gaussian in figure 2).\n3. The distribution of $u_c$ learned by CNNs is not discussed.\n4. The reviewer suggests the authors empirically analyze some more CNNs optimized for different tasks to see whether the results hold across a wide range of tasks.\n\nMinor comments:\nIt is a very well written paper, and the reviewer enjoys it during the review.  \n  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "gt8hTszN71c", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Review\n\n### Summary\n\nThe authors propose an alternative explanation to the fact that Convolutional Neural Networks (CNNs) learn oriented bandpass filters. They suggest that it is due to the architecture of neural network and not the structure of natural images.\n\nThe two main contributions are:\n1. the alternative explanation\n2. empirical support that oriented bandpass filters are learned at each layer\n\n### Strengths\n\n* The paper is well written and easy to follow.\n* This is a good summary of Linear Operators and Fourier Analysis for non-expert\n\n### Weaknesses\n\n* To me, the contributions are not real. The paper intends at formulating a problem where there is none: the theory is known already and empirical evidences already exist (the repeated observations: yes CNN learns oriented bandpass filters and it is not a surprise). \n* It mixes contributions such as Olshausen and Field (1996), Karklin and Lewicki (2009) with the litterature of CNNs and the theoretical work of Bruna and Mallat (2013). The first two papers show how localized oriented bandpass filters emerge from unsupervised learning of a non-convolutional dictionary under sparsity constraints. This is a strong indication that these localized bandpass filters are appropriate to represent natural images and a strong confirmation for the use of convolution (a translation invariant representation). It appear that this is also what we observe when evaluating neuron receptive fields. Finally, the work of Bruna and Mallat pushes further the idea of invariance, trying to find a theoretical explanation of why stacking Linear and Non-linear operation provide an efficient representation. They build on wavelet transforms (oriented bandpass filters) because it is a translation invariant representation known to be sparse.\n* The theory is not rigorously presented. This is a good enough overview of a graduate course on Fourier analysis and linear operators.\n* Empirical evidences are not convincing. How would you get if you were fitting random 3x3 filters with oriented bandpass filters ?\n\n### Minor comments\n\n* The text in the figures is too small...\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers raised a number of concerns, but 
the authors provided no rebuttal to the reviewers' comments.

One reviewer felt the experimental fitting was not thorough enough.
Suppose one used layers of oriented bandpass filters, separated by
non-linearities, would that perform well on the task convnets are
trained on?

The AC doesn't agree with the arguments of R3.  I hope the comments of
the reviewers, particularly the many specific comments of reviewers R1 and
R2, will be helpful to you as you revise the manuscript.

The AC feels a more thorough experimental evaluation, and following-up
on many of the suggestions of the reviewers will lead to a strong
paper.  As it stands, however, with 3 recommendations for rejection (1
weak), and only 1 weak recommendation for acceptance, we need to reject.

Decision: Reject