The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods

1 INTRODUCTION . Understanding the success of deep convolutional neural networks on images remains challenging because images are high-dimensional signals and deep neural networks are highly-non linear models with a substantial amount of parameters : yet , the curse of dimensionality is seemingly avoided by these models . This problem has received a plethora of interest from the machine learning community . One approach taken by several authors ( Mairal , 2016 ; Li et al. , 2019 ; Shankar et al. , 2020 ; Lu et al. , 2014 ) has been to construct simpler models with more tractable analytical properties ( Jacot et al. , 2018 ; Rahimi and Recht , 2008 ) , that still share various elements with standard deep learning models . Those simpler models are based on kernel methods with a particular choice of kernel that provides a convolutional representation of the data . In general , these methods are able to achieve reasonable performances on the CIFAR-10 dataset . However , despite their simplicity compared to deep learning models , it remains unclear which ones of the multiple ingredients they rely on are essential . Moreover , due to their computational cost , it remains open to what extend they achieve similar performances on more complex datasets such as ImageNet . In this work , we show that an additional implicit ingredient , common to all those methods , consists in a data-dependent feature extraction step that makes the convolutional kernel data-driven ( as opposed to purely handcrafted ) and is key for obtaining good performances . Data driven convolutional kernels compute a similarity between two images x and y , using both their translation invariances and statistics from the training set of images X . In particular , we focus on similarities K that are obtained by first standardizing a representation Φ of the input images and then feeding it to a predefined kernel k : Kk , Φ , X ( x , y ) = k ( LΦx , LΦy ) , ( 1 ) where a rescaling and shift is ( potentially ) performed by a diagonal affine operator L = L ( Φ , X ) and is mainly necessary for the optimization step Jin et al . ( 2009 ) : it is typically a standardization . The kernel K ( x , y ) is said to be data-driven if Φ depends on training set X , and data-independent otherwise . This , for instance , is the case if a dictionary is computed from the data ( Li et al. , 2019 ; Mairal , 2016 ; Mairal et al. , 2014 ) or a ZCA ( Shankar et al. , 2020 ) is incorporated in this representation . The convolutional structure of the kernel K can come either from the choice of the representation Φ ( convolutions with a dictionary of patches ( Coates et al. , 2011 ) ) or by design of the predefined kernel k ( Shankar et al. , 2020 ) , or a combination of both ( Li et al. , 2019 ; Mairal , 2016 ) . One of the goal of this paper is to clearly state that kernel methods for vision do require to be data-driven and this is explicitly responsible for their success . We thus investigate , to what extent this common step is responsible for the success of those methods , via a shallow model . Our methodology is based on ablation experiments : we would like to measure the effect of incorporating data , while reducing other side effects related to the design of Φ , such as the depth of Φ or the implicit bias of a potential optimization procedure . Consequently , we focus on 1-hidden layer neural networks of any widths , which have favorable properties , like the ability to be a universal approximator under non-restrictive conditions . The output linear layer shall be optimized for a classification task , and we consider first layers which are predefined and kept fixed , similarly to Coates et al . ( 2011 ) . We will see below that simply initializing the weights of the first layer with whitened patches leads to a significant improvement of performances , compared to a random initialization , a wavelet initialization or even a learning procedure . This patch initialization is used by several works ( Li et al. , 2019 ; Mairal , 2016 ) and is implicitly responsible for their good performances . Other works rely on a whitening step followed by very deep kernels ( Shankar et al. , 2020 ) , yet we noticed that this was not sufficient in our context . Here , we also try to understand why incorporating whitened patches is helpful for classification . Informally , this method can be thought as one of the simplest possible in the context of deep convolutional kernel methods , and we show that the depth or the non-linearities of such kernels play a minor role compared to the use of patches . In our work , we decompose and analyze each step of our feature design , on gold-standard datasets and find that a method based solely on patches and simple non-linearities is actually a strong baseline for image classification . We investigate the effect of patch-based pre-processing for image classification through a simple baseline representation that does not involve learning ( up to a linear classifier ) on both CIFAR-10 and ImageNet datasets : the path from CIFAR-10 to ImageNet had never been explored until now in this context . Thus , we believe our baseline to be of high interest for understanding ImageNet ’ s convolutional kernel methods , which almost systematically rely on a patch ( or descriptor of a patch ) encoding step . Indeed , this method is straightforward and involves limited ad-hoc feature engineering compared to deep learning approach : here , contrary to ( Mairal , 2016 ; Coates et al. , 2011 ; Recht et al. , 2019 ; Shankar et al. , 2020 ; Li et al. , 2019 ) we employ modern techniques that are necessary for scalability ( from thousands to million of samples ) but can still be understood through the lens of kernel methods ( e.g. , convolutional classifier , data augmentation , ... ) . Our work allows to understand the relative improvement of such encoding step and we show that our method is a challenging baseline for classification on Imagenet : we outperform by a large margin the classification accuracy of former attempts to get rid of representation learning on the large-scale ImageNet dataset . While the literature provides a detailed analysis of the behavior of a dictionary of patches for image compression ( Wallace , 1992 ) , texture synthesis ( Efros and Leung , 1999 ) or image inpainting ( Criminisi et al. , 2004 ) , we have a limited knowledge and understanding of it in the context of image classification . The behavior of those dictionaries of patches in some classification methods is still not well understood , despite often being the very first component of many classic vision pipelines ( Perronnin et al. , 2010 ; Lowe , 2004 ; Oyallon et al. , 2018b ) . Here , we proposed a refined analysis : we define a Euclidean distance between patches and we show that the decision boundary between image classes can be approximated using a rough description of the image patches neighborhood : it is implied for instance by the fame low-dimensional manifold hypothesis ( Fefferman et al. , 2016 ) . Our paper is structured as follows : first , we discuss the related works in Sec . 2 . Then , Sec . 3 explains precisely how our visual representation is built . In Sec . 4 , we present experimental results on the vision datasets CIFAR-10 and the large scale ImageNet . The final Sec . 4.3 is a collection of numerical experiments to understand better the dictionary of patches that we used . Our code as well as commands to reproduce our results are available here : https : //github.com/louity/patches . 2 RELATED WORK . The seminal works by Coates et al . ( 2011 ) and Coates and Ng ( 2011 ) study patch-based representations for classification on CIFAR-10 . They set the first baseline for a single-layer convolutional network initialized with random patches , and they show it can achieve a non-trivial performance ( ∼ 80 % ) on the CIFAR-10 dataset . Recht et al . ( 2019 ) published an implementation of this technique and conducted numerous experiments with hundreds of thousands of random patches , improving the accuracy ( ∼ 85 % ) on this dataset . However , both works lack two key ingredients : online optimization procedure ( which allows to scale up to ImageNet ) and well-designed linear classifier ( as we propose a factorization of our linear classifier ) . Recently , ( Li et al. , 2019 ; Shankar et al. , 2020 ) proposed to handcraft kernels , combined with deep learning tools , in order to obtain high-performances on CIFAR-10 . Those performances match standard supervised methods ( ∼ 90 % ) which involve end-to-end learning of deep neural networks . Note that the line of work ( Li et al. , 2019 ; Shankar et al. , 2020 ; Mairal , 2016 ) employs a wellengineered combination of patch-extracted representation and a cascade of kernels ( possibly some neural tangent kernels ) . While their works suggest that patch extraction is crucial , the relative improvement due to basic-hyper parameters such as the number of patches or the classifier choice is unclear , as well as the limit of their approach to more challenging dataset . We address those issues . From a kernel methods perspective , a dictionary of random patches can be viewed as the building block of a random features method ( Rahimi and Recht , 2008 ) that makes kernel methods computationally tractable . Rudi et al . ( 2017 ) provided convergence rates and released an efficient implementation of such a method . However , previously mentioned kernel methods ( Mairal , 2016 ; Li et al. , 2019 ; Shankar et al. , 2020 ) have not been tested on ImageNet to our knowledge . Simple methods involving solely a single-layer of features have been tested on the ImageNet-2010 dataset1 , using for example SIFT , color histogram and Gabor texture encoding of the image with K-nearest neighbors , yet there is a substential gap in accuracy that we attempt to fill in this work on ImageNet-2012 ( or simply ImageNet ) . We note also that CNNs with random weights have been tested on ImageNet , yielding to low accuracies ( ∼ 20 % top-1 , ( Arandjelovic et al. , 2017 ) ) . The Scattering Transform ( Mallat , 2012 ) is also a deep non-linear operator that does not involve representation learning , which has been tested on ImageNet ( ∼ 45 % top-5 accuracy ( Zarka et al. , 2019 ) and CIFAR-10 ( ∼ 80 % , ( Oyallon and Mallat , 2015 ) ) and is related to the HoG and SIFT transforms ( Oyallon et al. , 2018a ) . Some works also study directly patch encoders that achieve competitive accuracy on ImageNet but involve deep cascade of layers that are difficult to interpret ( Oyallon et al. , 2017 ; Zarka et al. , 2019 ; Brendel et al. , 2019 ) . Here , we focus on shallow classifiers . 3 METHOD . We first introduce our preliminary notations to describe an image . A patch p of size P of a larger image x , is a restriction of that image to a squared domain of surface P 2 . We denote by N2 the size of the natural image x and require that P ≤ N . Hence , for a spatial index i of the image , pi , x represents the patch of image x located at i . We further introduce the collection of all overlapping patches of that image , denoted by : Px = { pi , x , i ∈ I } where I is a spatial index set such that |I| = ( N − P + 1 ) 2 . Fig . 1 corresponds to an overview of our classification pipeline that consist of 3 steps : an initial whitening step of a dictionary D of random patches , followed by a nearest neighbor quantization of images patches via D that are finally spatially averaged . 1As one can see on the Imagenet2010 leaderboard http : //image-net.org/challenges/LSVRC/2010/results , and the accuracies on ImageNet2010 and ImageNet2012 are comparable . Whitening We describe the single pre-processing step that we used on our image data , namely a whitening procedure on patches . Here , we view natural image patches of size P 2 as samples from a random vector of mean µ and covariance Σ . We then consider whitening operators which act at the level of each image patch by first subtracting its mean µ then applying the linear transformation W = ( λI + Σ ) −1/2 to the centered patch . The additional whitening regularization with parameter λ was used to avoid ill-conditioning effects . Figure 1 : Our classification pipeline described synthetically to explain how we build the representation Φ ( x ) of an input image x . Dictionary … Find Q-nearest neighbours per patch 0 1patch 1 patch 2 … 01 … 1 1 … 01 … 1 … Spatial Average pooling … 1 1 … … … … 01 … 1 …2 0 2 2 1 1 2 … … 2 0 … 2 … patch 1 patch 2 input x Split the image in overlapping patches Representation Dictionary ( x ) < latexit sha1_base64= '' TGV8Qz6p5CvLBilkjVUuHjjuCuo= '' > AAACKnicbZDLSsNAFIYn9VbrLerSTbAIFaRMSsF2V3TjsoK9QBPKZDpph04uzEzEGvIaPoWP4FYfwF1xqQ/iJM3Ctv4w8PGfc+YcfidkVEgI51phY3Nre6e4W9rbPzg80o9PuiKIOCYdHLCA9x0kCKM+6UgqGemHnCDPYaTnTG/Teu+RcEED/0HOQmJ7aOxTl2IklTXUYWxlnwz42LFjWK03TFgzrzJQygA2m83Eak9o5ekyGeplWIWZjHUwcyiDXO2h/m2NAhx5xJeYISEGJgylHSMuKWYkKVmRICHCUzQmA4U+8oiw4+yoxLhQzshwA66eL43M/TsRI0+ImeeoTg/JiVitpeZ/tUEk3YYdUz+MJPHxYpEbMUMGRhqTMaKcYMlmChDmVN1q4AniCEsV5tIWSafPaSrmagbr0K1VTcX39XLrJs+nCM7AOagAE1yDFrgDbdABGLyAN/AOPrRX7VOba1+L1oKWz5yCJWk/v7NTo90= < /latexit > < latexit sha1_base64= '' TGV8Qz6p5CvLBilkjVUuHjjuCuo= '' > AAACKnicbZDLSsNAFIYn9VbrLerSTbAIFaRMSsF2V3TjsoK9QBPKZDpph04uzEzEGvIaPoWP4FYfwF1xqQ/iJM3Ctv4w8PGfc+YcfidkVEgI51phY3Nre6e4W9rbPzg80o9PuiKIOCYdHLCA9x0kCKM+6UgqGemHnCDPYaTnTG/Teu+RcEED/0HOQmJ7aOxTl2IklTXUYWxlnwz42LFjWK03TFgzrzJQygA2m83Eak9o5ekyGeplWIWZjHUwcyiDXO2h/m2NAhx5xJeYISEGJgylHSMuKWYkKVmRICHCUzQmA4U+8oiw4+yoxLhQzshwA66eL43M/TsRI0+ImeeoTg/JiVitpeZ/tUEk3YYdUz+MJPHxYpEbMUMGRhqTMaKcYMlmChDmVN1q4AniCEsV5tIWSafPaSrmagbr0K1VTcX39XLrJs+nCM7AOagAE1yDFrgDbdABGLyAN/AOPrRX7VOba1+L1oKWz5yCJWk/v7NTo90= < /latexit > < latexit sha1_base64= '' TGV8Qz6p5CvLBilkjVUuHjjuCuo= '' > AAACKnicbZDLSsNAFIYn9VbrLerSTbAIFaRMSsF2V3TjsoK9QBPKZDpph04uzEzEGvIaPoWP4FYfwF1xqQ/iJM3Ctv4w8PGfc+YcfidkVEgI51phY3Nre6e4W9rbPzg80o9PuiKIOCYdHLCA9x0kCKM+6UgqGemHnCDPYaTnTG/Teu+RcEED/0HOQmJ7aOxTl2IklTXUYWxlnwz42LFjWK03TFgzrzJQygA2m83Eak9o5ekyGeplWIWZjHUwcyiDXO2h/m2NAhx5xJeYISEGJgylHSMuKWYkKVmRICHCUzQmA4U+8oiw4+yoxLhQzshwA66eL43M/TsRI0+ImeeoTg/JiVitpeZ/tUEk3YYdUz+MJPHxYpEbMUMGRhqTMaKcYMlmChDmVN1q4AniCEsV5tIWSafPaSrmagbr0K1VTcX39XLrJs+nCM7AOagAE1yDFrgDbdABGLyAN/AOPrRX7VOba1+L1oKWz5yCJWk/v7NTo90= < /latexit > < latexit sha1_base64= '' TGV8Qz6p5CvLBilkjVUuHjjuCuo= '' > AAACKnicbZDLSsNAFIYn9VbrLerSTbAIFaRMSsF2V3TjsoK9QBPKZDpph04uzEzEGvIaPoWP4FYfwF1xqQ/iJM3Ctv4w8PGfc+YcfidkVEgI51phY3Nre6e4W9rbPzg80o9PuiKIOCYdHLCA9x0kCKM+6UgqGemHnCDPYaTnTG/Teu+RcEED/0HOQmJ7aOxTl2IklTXUYWxlnwz42LFjWK03TFgzrzJQygA2m83Eak9o5ekyGeplWIWZjHUwcyiDXO2h/m2NAhx5xJeYISEGJgylHSMuKWYkKVmRICHCUzQmA4U+8oiw4+yoxLhQzshwA66eL43M/TsRI0+ImeeoTg/JiVitpeZ/tUEk3YYdUz+MJPHxYpEbMUMGRhqTMaKcYMlmChDmVN1q4AniCEsV5tIWSafPaSrmagbr0K1VTcX39XLrJs+nCM7AOagAE1yDFrgDbdABGLyAN/AOPrRX7VOba1+L1oKWz5yCJWk/v7NTo90= < /latexit > D < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > D < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > < latexit sha1_base64= '' Rb/kpNpYCeioxAjDXIA8A4KhYjk= '' > AAACLnicbZDLSsNAFIYn9VbrLerSTbAILqQksfayK+rCZQXbCkkok+m0HTq5MDMRasiL+BQ+glt9AMGFuHDjYzhJs7CtPwx8/OecOYffDSnhQtc/lMLK6tr6RnGztLW9s7un7h90eRAxhDsooAG7dyHHlPi4I4ig+D5kGHouxT13cpXWew+YcRL4d2IaYseDI58MCYJCWn21GtvZJxYbuU6sV8ya0ayaZ3rlot48rzUlNOt6wzQT24NijCCNr5Okr5b1ip5JWwYjhzLI1e6r3/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ== < /latexit > … The whitening operation is defined up to an isometry , but the Euclidean distance between whitened patches ( i.e. , the Mahanobolis distance ( Chandra et al. , 1936 ) ) is not affected by the choice of such isometry ( choices leading to PCA , ZCA , ... ) , as discussed in Appendix A . In practice , the mean and covariance are estimated empirically from the training set to construct the whitening operators . For the sake of simplicity , we only consider whitened patches , and unless explicitly stated , we assume that each patch p is already whitened , which holds in particular for the collection of patches in Px of any image x . Once this whitening step is performed , the Euclidean distance over patches is approximatively isotropic and is used in the next section to represent our signals . Q-Nearest Neighbors on patches The basic idea of this algorithm is to compare the distances between each patch of an image and a fixed dictionary of patches D , with size |D| that is the number of patches extracted . Note that we also propose a variant where we simply use a soft-assignment operator . For a fixed dataset , this dictionary D is obtained by uniformly sampling patches from images over the whole training set . We augmentD into ∪d∈D { d , −d } because it allows the dictionary of patches to be contrast invariant and we observe it leads to better classification accuracies ; we still refer to it as D. An illustration is given by Fig . 2 . Once the dictionary D is fixed , for each patch pi , x we consider the set Ci , x of pairwise distances Ci , x = { ‖pi , x − d‖ , d ∈ D } . For each whitened patch we encode the Q-Nearest Neighbors of pi , x from the set D , for some Q ∈ N. More formally , we consider τi , x the Q-th smallest element of Ci , x , and we define the Q-Nearest Neighbors binary encoding as follow , for ( d , i ) ∈ D × I : φ ( x ) d , i = { 1 , if ‖pi , x − d‖ ≤ τi , x 0 , otherwise . ( 2 ) Eq . 2 can be viewed as a Vector Quantization ( VQ ) step with hard-assignment ( Coates and Ng , 2011 ) . The representation φ encodes the patch neighborhood in a subset of randomly selected patches and can be seen as a crude description of the topological geometry of the image patches . Moreover , it allows to view the distance between two images x , y as a Hamming distance between the patches neighborhood encoding as : ‖φ ( x ) − φ ( y ) ‖2 = ∑ i , d 1φ ( x ) d , i 6=φ ( y ) d , i . ( 3 ) In order to reduce the computational burden of our method , we perform an intermediary averagepooling step . Indeed , we subdivide I in squared overlapping regions Ij ⊂ I , leading to the representation Φ defined , for d ∈ D , j by : Φ ( x ) d , j = ∑ i∈Ij φ ( x ) d , i . ( 4 ) Hence , the resulting kernel is simply given by K ( x , y ) = 〈Φ ( x ) , Φ ( y ) 〉 . Implementation details can be found in Appendix B . The next section describes our classification pipeline , as we feed our representation Φ to a linear classifier on challenging datasets .

Summary: This paper proposes a powerful non-learning Kernal based baseline for ImageNet classification. The proposed non-learning Kernal based baseline (which can be interpretable to a vector quantization) shows comparable results (88.5) with AlexNet (89.1) in CIFAR-10 top-1 accuracy. The ImageNet result (39.4) shows that it is still challenging to classify the images without deep features, but about 40% is an impressive baseline without any learning method (e.g., these results is almost comparable to BagNet top-5 error).