Why Convolutional Networks Learn Oriented Bandpass Filters: Theory and Empirical Support

It has been repeatedly observed that convolutional architectures when applied to image understanding tasks learn oriented bandpass filters . A standard explanation of this result is that these filters reflect the structure of the images that they have been exposed to during training : Natural images typically are locally composed of oriented contours at various scales and oriented bandpass filters are matched to such structure . We offer an alternative explanation based not on the structure of images , but rather on the structure of convolutional architectures . In particular , complex exponentials are the eigenfunctions of convolution . These eigenfunctions are defined globally ; however , convolutional architectures operate locally . To enforce locality , one can apply a windowing function to the eigenfunctions , which leads to oriented bandpass filters as the natural operators to be learned with convolutional architectures . From a representational point of view , these filters allow for a local systematic way to characterize and operate on an image or other signal . We offer empirical support for the hypothesis that convolutional networks learn such filters at all of their convolutional layers . While previous research has shown evidence of filters having oriented bandpass characteristics at early layers , ours appears to be the first study to document the predominance of such filter characteristics at all layers . Previous studies have missed this observation because they have concentrated on the cumulative compositional effects of filtering across layers , while we examine the filter characteristics that are present at each layer . 1 INTRODUCTION . 1.1 MOTIVATION . Convolutional networks ( ConvNets ) in conjunction with deep learning have shown state-of-the-art performance in application to computer vision , ranging across both classification ( e.g. , Krizhevsky et al . ( 2012 ) ; Tran et al . ( 2015 ) ; Ge et al . ( 2019 ) ) and regression ( e.g. , Szegedy et al . ( 2013 ) ; Eigen & Fergus ( 2015 ) ; Zhou et al . ( 2017 ) ) tasks . However , understanding of how these systems achieve their remarkable results lags behind their performance . This state of affairs is unsatisfying not only from a scientific point of view , but also from an applications point of view . As these systems move beyond the lab into real-world applications better theoretical understanding can help establish performance bounds and increase confidence in deployment . Visualization studies of filters that have been learned during training have been one of the key tools marshalled to lend insight into the internal representations maintained by ConvNets in application to computer vision , e.g. , Zeiler & Fergus ( 2014 ) ; Yosinski et al . ( 2015 ) ; Mahendran & Vedaldi ( 2015 ) ; Shang et al . ( 2016 ) ; Feichtenhofer et al . ( 2018 ) . Here , an interesting repeated observation is that early layers in the studied networks tend to learn oriented bandpass filters , both in two image spatial dimenstions , ( x , y ) > , in application to single image analysis as well as in three spatiotemporal dimensions , ( x , y , t ) > , in application to video . An example is shown in Fig . 1 . Emergence of such filters seems reasonable , because local orientation captures the first-order correlation structure of the data , which provides a reasonable building block for inferring more complex structure ( e.g. , local measurements of oriented structure can be assembled into intersections to capture corner structure , etc. ) . Notably , however , more rigorous analyses of exactly why oriented bandpass filters might be learned has been limited . This state of affairs motivates the current paper in its argument that the analytic structure of ConvNets constrains them to learn oriented bandpass filters . 1.2 RELATED RESEARCH . Visualization of receptive field profiles ( i.e. , pointspread functions Lim ( 1990 ) ) of the convolutional filters learned by contemporary ConvNets is a popular tool for providing insight into the image properties that are being represented by a network . A notable trend across these studies is that early layers appear to learn oriented bandpass filters in both two spatial dimensions , e.g. , Zeiler & Fergus ( 2014 ) ; Springenberg et al . ( 2015 ) ; Yosinski et al . ( 2015 ) ; Shang et al . ( 2016 ) , as well as three spatiotemporal dimensions , e.g. , Feichtenhofer et al . ( 2018 ) . Indeed , earlier studies with architectures that also constrained their filters to be convolutional in nature , albeit using a Hebbian learning strategy MacKay ( 2003 ) rather than the currently dominant back-propagation approach Rumelhart et al . ( 1986 ) , also yielded filters that visualized as having oriented bandpass filter characteristics Linsker ( 1986 ) . Interestingly , biological vision systems also are known to show the presence of oriented bandpass filters at their earlier layers of processing in visual cortex ; see Hubel & Wiesel ( 1962 ) for pioneering work along these lines and for more general review DeValois & DeValois ( 1988 ) . The presence of oriented bandpass filters in biological systems often has been attributed to their being well matched to the statistics of natural images Field ( 1987 ) ; Olshausen & Field ( 1996 ) ; Karklin & Lewicki ( 2009 ) ; Simoncelli & Olshausen ( 2001 ) , e.g. , the dominance of oriented contours at multiple scales . Similar arguments have been made regarding why such filters are learned by ConvNets . Significantly , however , studies have shown that even when trained with images comprised of random noise patterns , convolutional architectures still learn oriented bandpass filters Linsker ( 1986 ) . These latter results suggest that the emergence of such filter tunings can not be solely attributed to systems being driven to learn filters that were matched to their training data . Similarly , recent work showed that randomly initialized networks serve well in image restoration problems Ulyanov et al . ( 2018 ) . Some recent multilayer convolutional architectures have specified their earliest layers to have oriented bandpass characteristics , e.g. , Bruna & Mallat ( 2013 ) ; Jacobsen et al . ( 2016 ) ; Hadji & Wildes ( 2017 ) ; indeed , some have specified such filters across all layers Bruna & Mallat ( 2013 ) ; Hadji & Wildes ( 2017 ) . These design decisions have been variously motivated in terms of being well matched to primitive image structure Hadji & Wildes ( 2017 ) or providing useful building blocks for learning higher-order structures Jacobsen et al . ( 2016 ) and capturing invariances Bruna & Mallat ( 2013 ) . Other work has noted that purely mathematical considerations show that ConvNets are well suited to realizing filter designs for capturing multiscale , windowed spectra Bruna et al . ( 2016 ) ; however , it did not explicitly established the relationship to eigenfunctions of convolution nor offer an explanation for why deep-learning yields oriented bandpass filters when applied to ConvNets . It also did not provide empirical investigation of exactly what filter characteristics are learned at each convolutional layer of ConvNets . 1.3 CONTRIBUTIONS . In the light of previous research , the present work appears to be the first to offer an explanation of why ConvNets learn oriented bandpass filters , independently of the input , by appeal to the inherent properties of their architectures in terms of their eigenfunctions . By definition , the convolutional layers of a ConvNet are governed by the properties of convolution . For present purposes , a key property is that the eigenfunctions of convolution are complex exponentials . Imposing locality on the eigenfunctions leads to oriented bandpass filters , which therefore are the appropriate filters to be learned by a ConvNet . Indeed , these theoretical considerations suggest that oriented bandpass filters should be learned at all layers of a ConvNet , not just at early layers . We provide empirical support for this observation by examining filters across all convolutional layers of three standard ConvNets ( AlexNet Krizhevsky et al . ( 2012 ) , ResNet He et al . ( 2016 ) and VGG16 Simonyan & Zisserman ( 2014 ) ) and show that both numerically and visually they are well characterized as having learned oriented bandpass filters at all their convolutional layers . Our empirical study is distinct from earlier visualization efforts , which concentrate on the cumulative compositional results of filtering across layers that typically show emergence of complicated structures in the layerwise feature maps , while we focus on the complementary question of what primitive filter characteristics have been learned at each individual layer and offer both numerical as well as visualization analyses . 2 THEORY . This section details a novel explanation for why ConvNets learn oriented bandpass filters . The first two subsections largely review standard material regarding linear systems theory Oppenheim et al . ( 1983 ) and related topics Kaiser ( 2011 ) ; Kusse & Westwig ( 2006 ) , but are necessary to motivate properly our explanation . The final subsection places the material in the context of ConvNets . 2.1 EIGENFUNCTIONS OF CONVOLUTION . Let L be a linear operator on a function space . The set of eigenfunctions φn associated with this operator satisfy the condition Kusse & Westwig ( 2006 ) Lφn = λnφn . ( 1 ) That is , the operator acts on the eigenfunctions simply via multiplication with a constant , λn , referred to as the eigenvalue . It sometimes also is useful to introduce a ( positive definite ) weighting function , w , which leads to the corresponding constraint Lφn = λnwφn . ( 2 ) For cases where any function in the space can be expanded as a linear sum of the eigenfunctions , it is said that the collection of eigenfunctions form a complete set . Such a set provides a convenient and canonical spanning representation . Let x = ( x1 , x2 , . . . , xn ) > , a = ( a1 , a2 , . . . , an ) > and u = ( u1 , u2 , . . . , un ) > . For the space of convolutions , with the convolution of two functions , f ( x ) and h ( x ) defined as f ( x ) ∗ h ( x ) = ∫ ∞ −∞ f ( x− a ) h ( a ) da ( 3 ) it is well known that functions of the form f ( x ) = eiu > x are eigenfunctions of convolution Oppenheim et al . ( 1983 ) , i.e. , ∫ ∞ −∞ eiu > ( x−a ) h ( a ) da = eiu > x ∫ ∞ −∞ e−iu > ah ( a ) da ( 4 ) with the equality achieved via appealing to eiu > ( x−a ) = eiu > xe−iu > a and subsequently factoring eiu > x outside the integral as it is independent of a . The integral on the right hand side of ( 4 ) , ∫ ∞ −∞ e−iu > ah ( a ) da , ( 5 ) is the eigenvalue , referred to as the modulation transfer function ( MTF ) in signal processing Oppenheim et al . ( 1983 ) . Noting that eiu > x = cos ( u > x ) + i sin ( u > x ) leads to the standard interpretation of u in terms of frequency of the function ( e.g. , input signal ) . Given the eigenfunctions of convolution are parameterized in terms of their frequencies , it is useful to appeal to the Fourier transform of function f ( x ) , where we use the form Horn ( 1986 ) F ( u ) = ∫ ∞ −∞ f ( x ) e−iu > x dx , ( 6 ) because any convolution can be represented in terms of how it operates via simple multiplication of the eigenvalues , ( 5 ) , with the eigenfunctions , eiu > x , with u given by ( 6 ) . Thus , this decomposition provides a canonical way to decompose f ( x ) and explicate how a convolution operates on it .

Summary: This paper presents an explanation of why convolutional neural networks learn oriented bandpass filters - as has been commonly shown for early layers in various ConvNet architectures. The main argument is that oriented bandpass filters are the eigen-functions of localized convolution operators and in order to span the input signal space (regardless of its a structure) the network will always learn these functions as filters. Additionally, because this result is independent of input signal it should happen at all layers. These are demonstrated by examining filter of several trained neural network architectures and fitting them to Gabors - showing the fit is good with low residual error.