Why Convolutional Networks Learn Oriented Bandpass Filters: A Hypothesis

1 INTRODUCTION . 1.1 MOTIVATION . Convolutional networks ( ConvNets ) in conjunction with deep learning have shown state-of-the-art performance in application to computer vision , ranging across both classification , e.g. , ( Krizhevsky et al. , 2012 ; Tran et al. , 2015 ; Ge et al. , 2019 ) and regression , e.g. , ( Szegedy et al. , 2013 ; Eigen & Fergus , 2015 ; Zhou et al. , 2017 ) tasks . However , understanding of how these systems achieve their remarkable results lags behind their performance . This state of affairs is unsatisfying not only from a scientific point of view , but also from an applications point of view . As these systems move beyond the lab into real-world applications better theoretical understanding can help establish performance bounds and increase confidence in deployment . Visualization studies of filters that have been learned during training have been one of the key tools marshaled to lend insight into the internal representations maintained by ConvNets in application to computer vision , e.g. , ( Zeiler & Fergus , 2014 ; Yosinski et al. , 2015 ; Mahendran & Vedaldi , 2015 ; Shang et al. , 2016 ; Feichtenhofer et al. , 2018 ) . Here , an interesting repeated observation is that early layers in the studied networks tend to learn oriented bandpass filters , both in two image spatial dimenstions , ( x , y ) > , in application to single image analysis as well as in three spatiotemporal dimensions , ( x , y , t ) > , in application to video . An example is shown in Figure 1 . Emergence of such filters seems reasonable , because local orientation captures the first-order correlation structure of the data , which provides a reasonable building block for inferring more complex structure ( e.g. , local measurements of oriented structure can be assembled into intersections to capture corner structure , etc. ) . Notably , however , more rigorous analyses of exactly why oriented bandpass filters might be learned has been limited . This state of affairs motivates the current paper in its argument that the analytic structure of ConvNets constrains them to learn oriented bandpass filters . 1.2 RELATED RESEARCH . Visualization of receptive field profiles ( i.e. , pointspread functions ( Lim , 1990 ) ) of the convolutional filters learned by contemporary ConvNets is a popular tool for providing insight into the image properties that are being represented by a network . A notable trend across these studies is that early layers appear to learn oriented bandpass filters in both two spatial dimensions , e.g. , ( Zeiler & Fergus , 2014 ; Springenberg et al. , 2015 ; Yosinski et al. , 2015 ; Shang et al. , 2016 ) as well as three spatiotemporal dimensions , e.g. , ( Feichtenhofer et al. , 2018 ) . Indeed , earlier studies with architectures that also constrained their filters to be convolutional in nature , albeit using a Hebbian learning strategy ( MacKay , 2003 ) rather than the currently dominant back-propagation approach ( Rumelhart et al. , 1986 ) , also yielded filters that visualized as having oriented bandpass filter characteristics ( Linsker , 1986 ) . Interestingly , biological vision systems also are known to show the presence of oriented bandpass filters at their earlier layers of processing in visual cortex ; see Hubel & Wiesel ( 1962 ) for pioneering work along these lines and for more general review DeValois & DeValois ( 1988 ) . The presence of oriented bandpass filters in biological systems often has been attributed to their being well matched to the statistics of natural images ( Field , 1987 ; Olshausen & Field , 1996 ; Karklin & Lewicki , 2009 ; Simoncelli & Olshausen , 2001 ) , e.g. , the dominance of oriented contours at multiple scales . Similar arguments have been made regarding why such filters are learned by ConvNets . Significantly , however , studies have shown that even when trained with images comprised of random noise patterns , convolutional architectures still learn oriented bandpass filters ( Linsker , 1986 ) . These later results suggest that the emergence of such filter tunings can not be solely attributed to systems being driven to learn filters that were matched to their training data . Interestingly , some recent multilayer convolutional architectures have specified their earliest layers to have oriented bandpass characteristics ( Bruna & Mallat , 2013 ; Jacobsen et al. , 2016 ; Hadji & Wildes , 2017 ) ; indeed , some have specified such filters across all layers ( Bruna & Mallat , 2013 ; Hadji & Wildes , 2017 ) . These design decisions have been variously motivated in terms of being well matched to primitive image structure ( Hadji & Wildes , 2017 ) or providing useful building blocks for learning higher-order structures ( Jacobsen et al. , 2016 ) and capturing invariances ( Bruna & Mallat , 2013 ) . Other work has noted that purely mathematical considerations show that ConNets are well suited to designs that capture multiscale , windowed spectra Bruna et al . ( 2016 ) ; however , it did not explictly established the relationship to eigenfunctions of convolution nor offered an explanation for why deep-learning yields oriented bandpass filters when applied to ConvNets . In the light of previous research , the present work appears to be the first to offer an explanation of why ConvNets learn oriented bandpass filters by appeal to the inherent properties of their architectures . By definition , the convolutional layers of a ConvNet are governed by the properties of convolution . For present purposes , a key property is that the eigenfunctions of convolution are complex exponentials . Imposing locality on the eigenfunctions leads to oriented bandpass filters , which therefore are the appropriate filters to be learned by a ConvNet . 2 ANALYTIC APPROACH . This section details a novel explanation for why ConvNets learn oriented bandpass filters . The first two subsections largely review standard material regarding linear systems theory ( Oppenheim et al. , 1983 ) and related topics ( Kaiser , 2011 ; Kusse & Westwig , 2006 ) , but are necessary to motivate properly our explanation . The final subsection places the material in the specific context of ConvNets . 2.1 EIGENFUNCTIONS OF CONVOLUTION . Let L be a linear operator on a function space . The set of eigenfunctions φn associated with this operator satisfy the condition ( Kusse & Westwig , 2006 ) Lφn = λnφn . ( 1 ) That is , the operator acts on the eigenfunctions simply via multiplication with a constant , λn , referred to as the eigenvalue . It sometimes also is useful to introduce a ( positive definite ) weighting function , w , which leads to the corresponding constraint Lφn = λnwφn . ( 2 ) For cases where any function in the space can be expanded as a linear sum of the eigenfunctions , it is said that the collection of eigenfunctions form a complete set . Such a set provides a convenient and canonical spanning representation . Let x = ( x1 , x2 , . . . , xn ) > , a = ( a1 , a2 , . . . , an ) > and u = ( u1 , u2 , . . . , un ) > . For the space of convolutions , with the convolution of two functions , f ( x ) and h ( x ) defined as f ( x ) ∗ h ( x ) = ∫ ∞ −∞ f ( x− a ) h ( a ) da ( 3 ) it is well known that functions of the form f ( x ) = eiu > x are eigenfunctions of convolution ( Oppenheim et al. , 1983 ) , i.e. , ∫ ∞ −∞ eiu > ( x−a ) h ( a ) da = eiu > x ∫ ∞ −∞ e−iu > ah ( a ) da ( 4 ) with the equality achieved via appealing to eiu > ( x−a ) = eiu > xe−iu > a and subsequently factoring eiu > x outside the integral as it is independent of a . The integral on the right hand side of ( 4 ) , ∫ ∞ −∞ e−iu > ah ( a ) da , ( 5 ) is the eigenvalue , referred to as the modulation transfer function ( MTF ) in signal processing ( Oppenheim et al. , 1983 ) . Noting that eiu > x = cos ( u > x ) + i sin ( u > x ) leads to the standard interpretation of u in terms of frequency of the function ( e.g. , input signal ) . Given the eigenfunctions of convolution are parameterized in terms of their frequencies , it is useful to appeal to the Fourier transform of function f ( x ) , where we use the form ( Horn , 1986 ) F ( u ) = ∫ ∞ −∞ f ( x ) e−iu > x dx , ( 6 ) because any convolution can be represented in terms of how it operates via simple multiplication of the eigenvectors , ( 5 ) , with the eigenfunctions , eiu > x , with u given by ( 6 ) . Thus , this decomposition provides a canonical way to decompose f ( x ) and explicate how a convolution operates on it . 2.2 IMPOSING LOCALITY . Understanding convolution purely in terms of its eigenvectors and eigenvalues provides only a global representation of operations , as notions of signal locality , x , are lost in the global transformation to the frequency domain , u . This state of affairs often is unsatisfactory from a representational point of view because one wants to understand the structure of the signal ( e.g. , an image ) on a more local basis ( e.g. , one wants to detect objects as well as their image coordinates ) . This limitation can be ameliorated by defining a windowed Fourier transform ( Kaiser , 2011 ) , as follows ( Jahne & Hausbecker , 2000 ) . Let w ( x ) be a windowing function that is positive valued , symmetric and monotonically decreasing from its center so as to provide greatest emphasis at its center . A Windowed Fourier Transform ( WFT ) of f ( x ) can then be defined as F ( uc , x ; w ) = ∫ ∞ −∞ f ( a ) w ( a− x ) e−iu > c a da . ( 7 ) Making use of the symmetry constraint that we have enforced on the windowing function allows for w ( x ) = w ( −x ) so that the WFT , ( 7 ) , can be rewritten as F ( uc , x ; w ) = ∫ ∞ −∞ f ( a ) w ( x− a ) eiu > c ( x−a ) e−iu > c x da , ( 8 ) which has the form of a convolution f ( x ) ∗ ( w ( x ) eiu > c x ) ( 9 ) with the inclusion of an additional phase component , e−iu > c x . To provide additional insight into the impact the WFT convolution , ( 9 ) , has on the function , f ( x ) , it is useful to examine the pointspread function , w ( x ) eiu > c x , in the frequency domain by taking its Fourier transform ( 6 ) , i.e. , calculate its MTF . We have∫ ∞ −∞ w ( x ) eiu > c xe−iu > x dx , ( 10 ) which via grouping by coefficients of x becomes∫ ∞ −∞ w ( x ) e−i ( u−uc ) > x dx . ( 11 ) Examination of ( 11 ) reveals that it is exactly the Fourier transform of the window function , cf . ( 6 ) , as shifted to the center frequencies , uc . Thus , operation of the WFT convolution , ( 9 ) , on a function , f ( x ) , passes central frequency , uc , relatively unattenuated , while it suppresses those that are further away from the central frequency according to the shape of the window function , w ( x ) , i.e. , it operates as a bandpass filter . Thus , convolution with a bank of such filters with varying central frequencies , uc , has exactly the desired result of providing localized measures of the frequency content of the function f ( x ) . Returning to the pointspread function itself , w ( x ) eiu > c x , and recalling that eiu > x = cos ( u > x ) + i sin ( u > c x ) , it is seen that in the signal domain , x , the filter will oscillate along the direction of uc while remaining relatively constant in the orthogonal direction , even as there is an overall amplitude fall-off with distance from the center according to the shape of w ( x ) , i.e. , we have an oriented bandpass filter . As a specific example ( Jahne & Hausbecker , 2000 ) , taking w ( x ) to be an n-dimensional Gaussianlike function , g ( x ; σ ) = κ e−‖x‖ 2/σ2 , with σ the standard deviation and κ a scaling factor , yields an n-dimensional Gabor-like filter , g ( x ; σ ) eiu > c x = g ( x ; σ ) ( cos ( u > c x ) + i sin ( u > c x ) ) , ( 12 ) which provides good joint localization of signal content in the signal and frequency domains Gabor ( 1946 ) . Indeed , visualization of these filters in two spatial dimensions ( Figure 2 ) provides strikingly similar appearance to those presented in Figure 1 , if in an idealized form . In particular , their pointspread functions oscillate according to a frequency ‖uc‖ along the direction , uc‖uc‖ , while remaining relatively constant in the orthogonal direction , even as there is an overall amplitude fall-off with distance from the center . In the frequency domain , they have peak power at uc with a fall-off following a Gaussian-like shape with standard deviation , 1/σ , that is the inverse of that used in specifying the window , w ( x ) . These observations hold because we already have seen , ( 11 ) , that the frequency domain representation of such a function is the Fourier transform of the window function , w ( x ) , shifted to the center frequencies , uc ; furthermore , the Fourier tranform of a function of the form g ( x ; σ ) has a similar form , albeit with an inverse standard deviation Bracewell ( 1986 ) .

Summary: This short, interesting paper provides a theoretical analysis to explain why we may expect to see bandpass oriented filters arise as a result of the convolutional structure of deep networks. The explanation boils down to the fact that the eigenfunctions of convolutions correspond to bandpass filters. This can explain surprising phenomena: bandpass filters arising when training on random noise inputs. The paper could have been even more interesting if some or more of the other questions had been considered: