Why Convolutional Networks Learn Oriented Bandpass Filters: A Hypothesis
Authors: Richard P. Wildes
Venue:  (2020)
Keywords: convolutional networks
computer vision
oriented bandpass filters
linear systems theory

Abstract: It has been repeatedly observed that convolutional architectures when applied to
image understanding tasks learn oriented bandpass filters. A standard explanation
of this result is that these filters reflect the structure of the images that they have
been exposed to during training: Natural images typically are locally composed
of oriented contours at various scales and oriented bandpass filters are matched
to such structure. The present paper offers an alternative explanation based not
on the structure of images, but rather on the structure of convolutional architectures.
In particular, complex exponentials are the eigenfunctions of convolution.
These eigenfunctions are defined globally; however, convolutional architectures
operate locally. To enforce locality, one can apply a windowing function to the
eigenfunctions, which leads to oriented bandpass filters as the natural operators
to be learned with convolutional architectures. From a representational point of
view, these filters allow for a local systematic way to characterize and operate on
an image or other signal.

help establish performance Under review as a conference paper at ICLR 2020
WHY CONVOLUTIONAL NETWORKS LEARN ORIENTED BANDPASS FILTERS: A HYPOTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
It has been repeatedly observed that convolutional architectures when applied image understanding tasks learn oriented bandpass filters. A standard explanation of this result that these filters reflect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass filters are matched to such structure. The present paper offers an alternative explanation based not on the structure of images, but rather on the structure of convolutional architec- the eigenfunctions of convolution.
tures. In particular, complex exponentials These eigenfunctions are defined globally; however, convolutional architectures operate locally. To enforce locality, can apply windowing function the eigenfunctions, which leads to oriented bandpass filters as the natural operators be learned with convolutional architectures. From a representational point of view, these filters allow for local systematic way to characterize and operate on an image or other signal.
INTRODUCTION
1.1 MOTIVATION
Convolutional networks (ConvNets) in conjunction with deep learning have shown state-of-the-art performance in application to computer vision, ranging across both classification, e.g.. (Krizhevsky et al. 2012, Tran et al., 2015; Ge et al. 2019) and regression, e.g., (Szegedy et al. 2013; Eigen & Fergus| 2015 Zhou et al. 2017) tasks. However, understanding of how these systems achieve their Zhou
remarkable results lags behind their performance. This state of affairs is unsatisfying not only from a scientific point of view, but also from an applications point of view. As these systems move beyond the lab into real-world applications better theoretical understanding bounds and increase confidence in deployment.
Visualization studies of filters that have been learned during training have been of the key tools marshaled to lend insight into the internal representations maintained by ConvNets in application to computer vision, e.g., (Zeiler & Fergus) 2014; Yosinski et al. 2015] Mahendran & Vedaldi 2015 Shang et al.) 2016 Feichtenhofer et al. 2018). Here, an interesting repeated observation is that early layers the studied networks tend to learn oriented bandpass filters, both in two image spatial dimenstions, (Ir, y) in application to single image analysis as well as in three spatiotemporal dimensions, (x, t)t TTT application video. An example shown in Figure 1. Emergence of such filters seems reasonable, because local orientation captures the first-order correlation structure of the data, which provides reasonable building block for inferring more complex structure (e.g.. local measurements of oriented structure can be assembled into intersections to capture corner structure, etc.). Notably, however, more rigorous analyses exactly why oriented bandpass filters might be learned has been limited. This state of affairs motivates the current paper in its argument that the analytic structure of ConvNets constrains them to learn oriented bandpass filters.
1.2 RELATED RESEARCH
Visualization of receptive field profiles (i.e.. pointspread functions (Lim 1990s) of the convolutional filters learned by contemporary ConvNets is a popular tool for providing insight into the image prop- erties that being represented by a network. A notable trend across these studies is that early layers

---

Under review as a conference paper at ICLR 2020
Figure I: Visualization of pointspread functions (convolutional kernels) observed to be learned in the early layers of ConvNets. Brightness corresponds to pointwise function values. The majority of the plots show characteristics of oriented bandpass filters in two spatial dimensions, i.e., oscillating values along one direction, while remaining relatively constant in the orthogonal direction, even as there is an overall amplitude fall-off with distance from the center. The specific examples derive from the early layers of a ResNet-50 architecture (He et. 2016) trained on ImageNet (Russakovsky et al. 2015
appear to learn oriented bandpass filters in both two spatial dimensions, e.g.. (Zeiler & Fergus 2014 Springenberg et al. 20155] Yosinski et al. 2015 Shang et al. (2016) as well as three spatiotemporal dimensions, e.g., (Feichtenhofer et al. [2018). Indeed, earlier studies with architectures that also con- strained their filters be convolutional in nature, albeit using Hebbian learning strategy (MacKay) 2003) rather than the currently dominant prok-propagttion approach (Rumelhart et al. 1986). also yielded filters that visualized as having oriented bandpass filter characteristics (Linsker 1986). In-
terestingly, biological vision systems also are known to show the presence of oriented bandpass filters their earlier layers of processing visual cortex; see Hubel Wiesel (1962) for pioneering work along these lines and for more general review Valoio & De Valois 19888).
The presence of oriented bandpass filters in biological systems often has been attributed to their be- ing well matched to the statistics of natural images (Field 1987 Olshausen & Field 1996 Karklin & Lewicki 2009; Simoncelli & Olshausen 2001), e.g., the dominance of oriented contours at mul- tiple scales. Similar arguments have been made regarding why such filters are learned by ConvNets. Significantly, however, studies have shown that even when trained with images comprised of ran- dom noise patterns, convolutional architectures still learn oriented bandpass filters (Linsker) 1986). These later results suggest that the emergence of such filter tunings cannot be solely attributed to systems being driven learn filters that were matched their training data.
Interestingly, some recent multilayer convolutional architectures have specified their earliest layers have oriented bandpass characteristics (Bruna & Mallat 2013) Jacobsen et al. 2016; Hadji & Wildes 2017); indeed, some have specified such filters across all layers (Bruna & Mallat [013]]
Hadji
Wildes 2017). These design decisions have been variously motivated in terms of being well
matched to primitive image structure (Hadji & Wildes 2017) providing useful building blocks for learning higher-order structures (Jacobsen et 2016 and capturing invariances (Bruna & Mallat, 2013). Other work has noted that purely mathematical considerations show that ConNets are well suited to designs that capture multiscale, windowed spectra Brunn et al. (2016); however,it did not explictly established the relationship to eigenfunctions of convolution nor offered explanation for why deep-learning yields oriented bandpass filters when applied to ConvNets.
In the light of previous research, the present work appears be the first offer explanation of why ConvNets learn oriented bandpass filters by appeal to the inherent propertics of their archi- tectures. By definition, the convolutional layers of ConvNet are governed by the properties of
2

---

Under review as a conference paper at ICLR 2020
convolution. For present purposes, key property is that the eigenfunctions of convolution are com- plex exponentials. Imposing locality on the eigenfunctions leads to oriented bandpass filters, which therefore are the appropriate filters to be learned by a ConvNet.
2 ANALYTIC APPROACH
This section details a novel explanation for why ConvNets learn oriented bandpass filters. The first two subsections largely review standard material regarding linear systems theory (Oppenheim et al., 1983] and related topics (Kaiser 2011: Kusse Westwig 2006), but necessary motivate prop- crly our explanation. The final subsection places the material in the specific context of ConvNets.
2.1 EIGENFUNCTIONS OF CONVOLUTION
Let C be linear operator on function space. The set of eigenfunctions on associated with this operator satisfy the condition (Kusse & Westwig 2006)
CØn Anan.
That is, the operator acts on the eigenfunctions simply via multiplication with a constant, An. re- ferred as the eigenvalue. It sometimes also is useful to introduce a (positive definite) weighting function, w, which leads to the corresponding constraint
Cin A, won.-
For cases where any function in the space can be expanded as a linear sum of the eigenfunctions, is said that the collection of cigenfunctions form complete set. Such set provides convenient and canonical spanning representation.
Let (I1, .2...... ................................ TT,)T, T. (a1, .2....,, a, and (u1, u2...... un)) T For the space of convolutions, with the convolution of two functions, f(x) and hxx) defined as
f(x) h(x) = / f(x- f(x-a)((a)) a)h(a)
(3)
it is well known that functions of the form f(x) = e:u are eigenfunctions of convolution Oppen- heim et 1983 i.e.
ii (x-a) da c in fo
0o
e cinu" --nuaaaa *h(a) da
(4)
with the equality achieved via appealing ( e xx-xxiii -u and subsequently factoring ciu? outside the integral as independent of a. The integral on the right hand side of (4
su (1(a) da,
5
the eigenvalue, referred to as the modulation transfer function (MTF) in signal processing (Oppen- heim et al. 1983). Noting that e cos(u u x) + isinnu x) leads to the standard interpretation of in terms of frequency of the function (e.g., input signal).
Given the eigenfunctions of convolution are parameterized in terms of their frequencies, it useful appeal to the Fourier transform of function f(x), where we use the form HHmm 1986)
F(u) = f -00
f(x)e- -tu" x dx,
(6)
because any convolution can represented in terms of how operates via simple multiplication of the eigenvectors, (5). with the eigenfunctions, ein 'x with u given by (6). Thus, this decomposition provides a canonical way to decompose f(x) and explicate how convolution operates on it.
3

---

Under review as a conference paper at ICLR 2020
2.2 IMPOSING LOCALITY
Understanding convolution purely in terms of its eigenvectors and eigenvalues provides only a global representation of operations, notions of signal locality, are lost in the global transformation to the frequency domain, u. This state of affairs often is unsatisfactory from representational point of view because one wants to understand the structure of the signal (e.g., an image) on a more local basis (e.g., one wants to detect objects as well as their image coordinates). This limitation can be ameliorated by defining windowed Fourier transform (aissr 2011), as follows (Jahne & Hausbecker 2000).
Let (xx be windowing function that positive valued, symmetric and monotonically decreasing from its center so as to provide greatest emphasis at its center. A Windowed Fourier Transform (WFT) of (x) can then be defined as
F(u,,x;w) { f(a)w((- x)e -u, x))- da.
Making use of the symmetry constraint that we have enforced on the windowing function allows for w(x) w(-x) so that the WFT, 7 can be rewritten as
/ f(a)w(( a) (x (x-n) -in da.
F(u x)
(8)
which has the form of a convolution
f(x) (wxx)etuXx)
(9)
with the inclusion of an additional phase component, e iu.
To provide additional insight into the impact the WFT convolution, 9). has on the function, f(x), it is useful examine the pointspread function, w(x)e""X* x in the frequency domain by taking its Fourier transform (6), i.e., calculate its MTF. We have
w (x)e -iu dx.
(10)
which via grouping by coefficients of becomes
wxx)e i(u uu) dxc
(11)
Examination of (IT) reveals that it is exactly the Fourier transform of the window function, cf. (6). shifted to the center frequencies, u, Thus, operation of the WFT convolution, (9). on function, fxx), passes central frequency, u,, relatively unattenuated, while suppresses those that are further away from the central frequency according to the shape of the window function, w(x),, i.., it operates as a bandpass filter. Thus, convolution with bank of such filters with varying central frequencies, ue, has exactly the desired result of providing localized measures of the frequency content of the function fx))
Returning to the pointspread function itself, w(x)e"M x and recalling that ein = cos(u x) + sin(u. x), is seen that in the signal domain, x, the filter will oscillate along the direction of u, while remaining relatively constant in the orthogonal direction, even as there is overall amplitude fall-off with distance from the center according to the shape of w(x), i.e., we have an oriented bandpass filter.
As specific example (Jahne & Hausbecker, 2000), taking wxx) to be an n-dimensional Gaussian- like function, g(x; σ) = K ||x||"/0" with a the standard deviation and и scaling factor, yields an n-dimensional Gabor-like filter.
g(x;a)e""]> g(x; o) (cos(u.) x) + sin(u
(12)
which provides good joint localization of signal content in the signal and frequency domains Gabor (1946). Indeed, visualization these filters in two spatial dimensions (Figure 2) provides strik- ingly similar appearance those presented in Figure if in idealized form. In particular, their
4

---

Under review as a conference paper at ICLR 2020
The left panel shows the pointspread function corresponding to the odd symmetry (sin) component, while the right panel shows its power in the frequency domain. Brightness corresponds to pointwise function values.
Figure 2:
Visualization of an analytically defined oriented bandpass filter 112
pointspread functions oscillate according to a frequency ||ue]] along the direction, [Tu:T while re- maining relatively constant in the orthogonal direction, even as there is an overall amplitude fall-off with distance from the center. the frequency domain, they have peak power at u, with a fall-off following a Gaussian-like like shape with standard deviation, 1/a, that is the inverse of that used in specifying the window, wx(x). These observations hold because we already have seen, (11), that the frequency domain representation of such a function is the Fourier transform of the window function, wxx), shifted to the center frequencies, uz; furthermore, the Fourier tranform of a function of the form (xx; σ) has similar form, albeit with inverse standard deviation Bracewell (9866).
u.
2.3 IMPLICATIONS FOR CONVNETS
Convolutions in ConvNets serve to filter the input signal to highlight its features according to the learned pointspread functions (convolutional kernels). Thus, convolution with the oriented filters shown in Figure 1 will serve highlight aspects of an image that are correspondingly oriented and at corresponding scales. The question at hand is, "Why did the ConvNet learn such filters?" The previous parts of this section have reviewed the fact that complex exponentials of the form ein cos(u" x) + sin(u" x) are the eigenfunctions of convolution. Thus, such frequency de- pendent functions similarly serve as the eigenfunctions of the convolutional operations in ConvNets. In particular, this result a basic property of the convolutional nature of the architecture, indepen- dent of the input the system. Thus, for any convolution in a ConvNet the frequency dependent eigenfunctions, g provide systematic way to represent their input.
As with the general discussion of locality presented in Subsection 2.2 for the specifics of ConvNets also is of interest to be able to characterize and operate locally on signal. At the level of convolu- tion, such processing is realized via pointspread functions that operate as bandpass filters, (9). Like any practical system, ConvNets will not capture continuous range of bandpass characteristics, as given by u, and the sampling will be limited by the number of filters the designer allows each layer, i.e., as a metaparameter of the system. Nevertheless, making use of these filters provides a systematic approach to representing the input signal.
Overall, the very convolutional nature of ConvNets inherently constrain and even define the filters that they learn, independent of their input or training. In particular, learning bandpass filters provides a canonical way to represent and operate on their input, as these serve as the localized eigenfunc- tions of convolution. As ConvNet is exposed is optimized by spanning much of the data as it can. Within the realm of convolution, in which ConvNet conv layers are defined, oriented bandpass filters provide the solution. They arise as the lo- cality constrained eigenfunctions of convolution and thereby have potential to provide span of any input signal in a localized manner. Thus, ConvNets are optimized by learning exactly such filters. Notably, since this explanation for why ConvNets learn oriented bandpass filters is independent of training data, it can explain why such filters emerge even when the training data lacks such pattern structure, including training random input signals, e.g., (Linsker 1986).
more and more training data, its representation
Two final implications suggest directions for future research. First, the analysis presented in this paper has been targeted toward an explanation for why ConvNets leam oriented bandpass filters in their early layers, as suggested in visualization studies discussed in Sec. [.2 However, our analy- sis of oriented bandpass filtes as the localized eigenfunctions of convolution is not specifie to early
5

---

the combined result across Under review as a conference paper at ICLR 2020
ConvNet layers. but rather applies to any convolutional layer. The result thereby raises the question of whether deeper ConvNet layers also learn oriented bandpass filters. Here, empirical data is lack- ing as visualization studies to date for any given layer concentrate all previous layers, rather than the filtering characteristics at just that layer. Thus, interesting future research in examining the filters at each layer in isolation from those at other layers is motivated by the analysis presented in the current paper. Along these lines, it is interesting to note that certain hand-crafted Nets already make use of the same set of oriented bandpass filters at all layers of the architecture and do so to good advantage in their targeted tasks of single image and video texture analysis (Bruna Mallat 2013 Hadji & Wildes 2017).
A second direction for future work involves use of parameterized filters during learning. In par- ticular, the presented theoretical motivation for learning of oriented bandpass operators as the con- volutional filters in ConvNets suggests the possibility of constraining the filters to be of that form in a learning-basse
framework. In such a framework, it would not be necessary for the training process to learn the numerical values for each and every individual filter value (i.e., each filter tap), but rather would merely need to learn a much smaller number of parameters, e.g., the values of the center frequency, u, and the standard deviation associated with the Gabor-like filter derived above (12). Such a constrained learning approach would require a much less intensive training procedure (e.g. involving far less data) compared to learnig values fo all individual filter taps, owing to the drastically reduced number of parameters that need to be estimated, even while being able to tune to the specifics of the task that is being optimized.
SUMMARY
Previous studies have demonstated that learned filters at the early layers of convolutional networks visualize as oriented bandpass filters. This phenomenon typically is explained via appeal to natural image statisties, i.e. natural images are dominated by oriented contours manifest across variety of scales and oriented bandpass filters are well matched to such structure. We have offered an alternative explanation in terms of the structure of convolutional networks themselves. Given that their convolutional layers necessarily operate within the space of convolutions, learning oriented bandpass filters provides the system with the potential to span possible input, even while preserving a notion of locality in the signal domain.
REFERENCES
R. Bracewell. The Fourier Trunsform and Its Applications. McGraw-Hill, NY, NY, 1986.
J. Bruna and Mallat. Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell., 35:1872-1886, 2013.
J. Bruna, S. Chintala, Y. LeCun, S. Piantino, A. Szlam, and M. Tygert. A matehatmical motivation for complex-valued convolutional networks. Neural Computation, 28:815-825, 2016.
R. De Valois and K. Delolois. Spatial Vision. Oxford University Press, NY, NY, 1988.
D. Eigen and R. Fergus. Predicting depth, surface normals and sematic labels with a common muli- scale convolutional architecture. In ICCV, 2015.
C. Feichtenhofer, A. Pinz, R. Wildes, and A. Zisserman. What have we learned from deep represen- tations for action recognition? In CVPR, 2018.
D. Field. Relations between the statistics natural images and the response properties of cortical cells. Journal of the Optical Society of America A, 4:2379, 1987.
D. Gabor. Theory of communication. Journal of the Institute of Electrical Engineers, 93:429-457, 1946.
W. Ge, X. Lin, and Y. Yu. Weakly supervised complementary parts models for fine- fine-grained image classification from the bottom up. In CVPR, 2019.
I. Hadji and R. Wildes. A spatiotemporal oriented energy network for dynamic texture recognition. In ICCV, 2017.

---

Under review as a conference paper at ICLR 2020
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
B. Horn. Robot Vision. MIT Press, Cambridge, MA, 1986.
D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 160:106 10-4044-44, 1962.
J H. Jacobsen, Gemert, Z. Lou, and A W.M. Smeulders. Structured receptive fields cnns. In CVPR, 2016.
B. Jahne and H. Hausbecker. Computer Vision and Applications. Academic Press, San Diego, CA, 2000.
G. Kaiser. A Friendly Guide to Wavelets. Modern Birkhauser Classics, Switzerland, 2011.
Y. Karklin and M. Lewicki. Emergence of complex cell properties by learning to generalize in natural scenes. Nature, 457:83-86 -3, 2009.
A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
B. Kusse and W. Westwig. Mathematical Physics. Wiley-VCHI Weinheim, FRG, 2006.
Lim. Two-Dimensional Signal and Image Processing. Prentice Hall, Upper Saddle River, NJ, 1990.
R. Linsker. From basic network principles to neural architecture. Proc. National Academy of Sci- ences USA, 83:7508-7512 7512, $390-8394, 8779,8783, 1986.
D. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, Cambridge, UK, 2003.
A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.
B. Olshausen and D. Field. Emergence of simple-cell field properties by learning sparse code for natural images. Nature, 381:6076 1996.
A. Oppenheim, A. Willsky, and I. Young. Signals and Systems. Prentice Hall, Upper Saddle River, NJ, 1983.
D. Rumelhart, G. Hinton, and R. Williams. Learning representations by back-propagating errors. Nature, 323:533-533 1986.
O. Russakovsky, Deng, H. Su, J. Krause, Satheesh, S. Ma, Huang, Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
W. Shang, K. Sohn, and H. Lee D. A. Enlitic. Understanding and improving convolutional neural networks via concatenated rectified linear units. In ICML, 2016.
Simoncelli and B. Olshausen. Natural image statisties and neural representation. Annual Review of Neurosccence, 24:1 193-1296., 2001.
Springenberg, A. Dosovitskiy, т. Brox, and M. Riedmiller. Striving for simplicity: The all convo- lutional net. In ICLR Workshop, 2015.
C. Szegedy, A. Toshev, and D. Erhan. Deep neural networks for object detection. In NIPS, 2013.
D. Tran, L. Bourdev, Fergus, Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. In ICML workshops, 2015.
M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. ECCV, 2014.
Zhou, M. Brown, N. Snavely, and D. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
7


Reviews:
[{"review_id": "SJlVT-tO9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper claims that convolutional filters in CNNs are not the result of fitting to the input data distribution but they are the optimal solution to a spectral decomposition of the convolutional operator. \n\nPositive things about this work:\n1) it is clearly written\n2) the fact that Gabor wavelets are the eigenfunctions of convolution is sound.\n3) it provides good food for thought about what needs to be learned and what comes from the pre- specified choice of architecture\n\nNegative things about this work:\n1) it misses references to relevant works. In particular, there is an article making essentially the same points:\nJoan Bruna, Soumith Chintala, Yann LeCun, Serkan Piantino, Arthur Szlam, and Mark Tygert, \"A mathematical motivation for complex-valued convolutional networks,\" Neural Computation, 28 (5): 815-825, 2016\nhttp://tygert.com/ccnet.pdf\nwhere these authors make the same conclusions and observe that learning reduces to figuring out the  windowing, number of scales, etc. but not the type of filters.\nThis prior work greatly reduces the impact of this contribution, unfortunately.\nThere are other references that are missing, but these are minor points compared to the above. For instance, I'd recommend to cite Hubel & Wiesel's seminal work on mapping the mammalian receptive fields, and older work by M. Lewicki about analyzing learned receptive fields by sparse coding algorithms, similarly to the cited B. Olshausen et al.  \n2) The Authors show that Gabor wavelets are eigenfunction of convolutions and they stop here to conclude that filters in CNNs are the way they are because of the architecture, but how about the effect of the non-linearities, depth and the type of cost used for training? The work is unfinished without a thorough analysis and discussion of these crucial aspects.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HygnHAMx5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Research Problem: Previous studies have showed that learned filters at the early layers of CNNs visualize as oriented bandpass filters. This phenomenon typically is explained via appeal to natural image statistics, i.e., natural images are dominated by oriented contours manifest across a variety of scales and oriented bandpass filters are well matched to such structure. \n\nContribution: This paper proposes an explanation in terms of the structure of convolutional networks themselves: Given that\ntheir convolutional layers necessarily operate within the space of convolutions, learning oriented bandpass filters provides the system with the potential to span possible input, even while preserving a notion of locality in the signal domain.\n\nQuestion: What is possible implication and applications of this  explanation in practice. It will be useful to add discussion and experimental results to show the advantage of this explanation in real applications.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BkxHYpOFcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a hypothesis on why neural network learns oriented bandpass filters. While most existing work attribute this phenomenon to image structures, this paper suggests that it might be a property of convolution. In particular, it shows Fourier basis are eigenfunctions of convolution, and band pass filters are eigenfunctions for the generalized eigen problem of convolution given a windowed weighting function, which corresponds to a windowed Fourier transform. \n\nThe mathematical observations are interesting, and the paper hypothesizes that this mathematical property encourages neural networks to learn oriented bandpass filters. However, it is unclear why the neural network should learn eigenfunctions as the filters. I understand the paper is proposing a hypothesis, but drawing a more solid conclusion is important. I am not recommending acceptance of this paper in the main conference, but it may be a good paper in a certain workshop.\n\nBesides, does higher layer of a deep neural network also learn bandpass filters w.r.t. its input feature map? How well the phenomenon and the hypothesis could generalize to the deeper layers?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkeI98tCYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This short, interesting paper provides a theoretical analysis to explain why we may expect to see bandpass oriented filters arise as a result of the convolutional structure of deep networks. The explanation boils down to the fact that the eigenfunctions of convolutions correspond to bandpass filters. This can explain surprising phenomena: bandpass filters arising when training on random noise inputs. The paper could have been even more interesting if some or more of the other questions had been considered:\n\n1. The effect of training on a specific dataset had been explained: how do the resulting filters change and how does that tie in with the theory?\n\n2. Some light shed on 3D filters in higher layers of the network.\n\n3. Whether any guiding principles emerge for conv. layer construction. \n ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an alternative explanation of the emergence of oriented bandpass filters in convolutional networks: rather than reflecting observed structure in images, these filters would be a consequence of the convolutional architecture itself and its eigenfunctions. 
Reviewers agree that the mathematical angle taken by the paper is interesting, however they also point out that crucial prior work making the same points exists, and that more thorough insights and analyses would be needed to make a more solid paper.
Given the closeness to prior work, we cannot recommend acceptance in this form.
Decision: Reject