Deep Network Classification by Scattering and Homotopy Dictionary Learning

1 INTRODUCTION . Deep convolutional networks have spectacular applications to classification and regression ( LeCun et al. , 2015 ) , but they are black boxes that are hard to analyze mathematically because of their architecture complexity . Scattering transforms are simplified convolutional neural networks with wavelet filters which are not learned ( Bruna & Mallat , 2013 ) . They provide state-of-the-art classification results among predefined or unsupervised representations , and are nearly as efficient as learned deep networks on relatively simple image datasets , such as digits in MNIST , textures ( Bruna & Mallat , 2013 ) or small CIFAR images ( Oyallon & Mallat , 2014 ; Mallat , 2016 ) . However , over complex datasets such as ImageNet , the classification accuracy of a learned deep convolutional network is much higher than a scattering transform or any other predefined representation ( Oyallon et al. , 2019 ) . A fundamental issue is to understand the source of this improvement . This paper addresses this question by showing that one can reduce the learning to a single dictionary matrix , which is used to compute a positive sparse ` 1 code . The resulting algorithm is implemented with a simplified convolutional neural network architecture illustrated in Figure 1 . The classifier input is a positive ` 1 sparse code of scattering coefficients calculated in a dictionary D. The matrix D is learned together with the classifier by minimizing a classification loss over a training set . We show that learning D improves the performance of a scattering representation considerably and is sufficient to reach a higher accuracy than AlexNet ( Krizhevsky et al. , 2012 ) over ImageNet 2012 . This cascade of well understood mathematical operators provides a simplified mathematical model to analyze optimization and classification performances of deep neural networks . Dictionary learning for classification was introduced in Mairal et al . ( 2009 ) and implemented with deep convolutional neural network architectures by several authors ( Sulam et al. , 2018 ; Mahdizadehaghdam et al. , 2019 ; Sun et al. , 2018 ) . To reach good classification accuracies , these networks cascade several dictionary learning blocks . As a result , there is no indication that these operators compute optimal sparse ` 1 codes . These architectures are thus difficult to analyze mathematically and involve heavy calculations . They have only been applied to small image classification problems such as MNIST or CIFAR , as opposed to ImageNet . Our architecture reaches a high classification performance on ImageNet with only one dictionaryD , because it is applied to scattering coefficients as opposed to raw images . Intra-class variabilities due to geometric image transformations such as translations or small deformations are linearized by a scattering transform ( Bruna & Mallat , 2013 ) , which avoids unnecessary learning . Learning a dictionary in a deep neural network requires to implement a sparse ` 1 code . We show that homotopy iterative thresholding algorithms lead to more efficient sparse coding implementations with fewer layers . We prove their exponential convergence in a general framework that includes the ALISTA ( Liu et al. , 2019 ) algorithm . The main contributions of the paper are summarized below : • A sparse scattering network architecture , illustrated in Figure 1 , where the classification is performed over a sparse code computed with a single learned dictionary of scattering coefficients . It outperforms AlexNet over ImageNet 2012 . • A new dictionary learning algorithm with homotopy sparse coding , optimized by gradient descent in a deep convolutional network . If the dictionary is sufficiently incoherent , the homotopy sparse coding error is proved to convergence exponentially . We explain the implementation and mathematical properties of each element of the sparse scattering network . Section 2 briefly reviews multiscale scattering transforms . Section 3 introduces homotopy dictionary learning for classification , with a proof of exponential convergence under appropriate assumptions . Section 4 analyzes image classification results of sparse scattering networks on ImageNet 2012 . 2 SCATTERING TRANSFORM . A scattering transform is a cascade of wavelet transforms and ReLU or modulus non-linearities . It can be interpreted as a deep convolutional network with predefined wavelet filters ( Mallat , 2016 ) . For images , wavelet filters are calculated from a mother complex wavelet ψ whose average is zero . It is rotated by r−θ , dilated by 2j and its phase is shifted by α : ψj , θ ( u ) = 2 −2jψ ( 2−jr−θu ) and ψj , θ , α = Real ( e−iα ψj , θ ) We choose a Morlet wavelet as in Bruna & Mallat ( 2013 ) to produce a sparse set of non-negligible wavelet coefficients . A ReLU is written ρ ( a ) = max ( a , 0 ) . Scattering coefficients of orderm = 1 are computed by averaging rectified wavelet coefficients with a subsampling stride of 2J : Sx ( u , k , α ) = ρ ( x ? ψj , θ , α ) ? φJ ( 2 Ju ) with k = ( j , θ ) where φJ is a Gaussian dilated by 2J ( Bruna & Mallat , 2013 ) . The averaging by φJ eliminates the variations of ρ ( x ? ψj , θ , α ) at scales smaller than 2J . This information is recovered by computing their variations at all scales 2j ′ < 2J , with a second wavelet transform . Scattering coefficients of order two are : Sx ( u , k , k′ , α , α′ ) = ρ ( ρ ( x ? ψj , θ , α ) ? ψj′ , θ′ , α′ ) ? φJ ( 2 Ju ) with k , k′ = ( j , θ ) , ( j′ , θ′ ) To reduce the dimension of scattering vectors , we define phase invariant second order scattering coefficients with a complex modulus instead of a phase sensitive ReLU : Sx ( u , k , k′ ) = ||x ? ψj , θ| ? ψj′ , θ′ | ? φJ ( 2Ju ) for j′ > j The scattering representation includes order 1 coefficients and order 2 phase invariant coefficients . In this paper , we choose J = 4 and hence 4 scales 1 ≤ j ≤ J , 8 angles θ and 4 phases α on [ 0 , 2π ] . Scattering coefficients are computed with the software package Kymatio ( Andreux et al. , 2018 ) . They preserve the image information , and x can be recovered from Sx ( Oyallon et al. , 2019 ) . For computational efficiency , the dimension of scattering vectors can be reduced by a factor 6 with a linear operator L that preserves the ability to recover a close approximation of x from LSx . The dimension reduction operator L of Figure 1 may be an orthogonal projection over the principal directions of a PCA calculated on the training set , or it can be optimized by gradient descent together with the other network parameters . The scattering transform is Lipschitz continuous to translations and deformations ( Mallat , 2012 ) . Intra-class variabilities due to translations smaller than 2J and small deformations are linearized . Good classification accuracies are obtained with a linear classifier over scattering coefficients in image datasets where translations and deformations dominate intra-class variabilities . This is the case for digits in MNIST or texture images ( Bruna & Mallat , 2013 ) . However , it does not take into account variabilities of pattern structures and clutter which dominate complex image datasets . To remove this clutter while preserving class separation requires some form of supervised learning . The sparse scattering network of Figure 1 computes a sparse code of scattering representation β = LSx in a learned dictionary D of scattering features , which minimizes the classification loss . For this purpose , the next section introduces a homotopy dictionary learning algorithm , implemented in a small convolutional network . 3 HOMOTOPY DICTIONARY LEARNING FOR CLASSIFICATION . Task-driven dictionary learning for classification with sparse coding was proposed in Mairal et al . ( 2011 ) . We introduce a small convolutional network architecture to implement a sparse ` 1 code and learn the dictionary with a homotopy continuation on thresholds . The next section reviews dictionary learning for classification . Homotopy sparse coding algorithms are studied in Section 3.2 . 3.1 SPARSE CODING AND DICTIONARY LEARNING . Unless specified , all norms are Euclidean norms . A sparse code approximates a vector β with a linear combination of a minimum number of columns Dm of a dictionary matrix D , which are normalized ‖Dm‖ = 1 . It is a vector α0 of minimum support with a bounded approximation error ‖Dα0 − β‖ ≤ σ . Such sparse codes have been used to optimize signal compression ( Mallat & Zhang , 1993 ) and to remove noise , to solve inverse problems in compressed sensing ( Candes et al. , 2006 ) , and for classification ( Mairal et al. , 2011 ) . In this case , the dictionary learning optimizes the matrix D in order to minimize the classification loss . The resulting columns Dm can be interpreted as classification features selected by the sparse code α0 . To enforce this interpretation , we impose that sparse code coefficients are positive , α0 ≥ 0 . Positive sparse coding Minimizing the support of a code α amounts to minimizing its ` 0 `` norm '' , which is not convex . This non-convex optimization is convexified by replacing the ` 0 norm by an ` 1 norm . Since α ≥ 0 , we have ‖α‖1 = ∑ m α ( m ) . The minimization of ‖α‖1 with ‖Dα− β‖ ≤ σ is solved by minimizing a convex Lagrangian with a multiplier λ∗ which depends on σ : α1 = argmin α≥0 1 2 ‖Dα− β‖2 + λ∗ ‖α‖1 ( 1 ) One can prove ( Donoho & Elad , 2006 ) that α1 ( m ) has the same support as the minimum support sparse code α0 ( m ) along m if the support size s and the dictionary coherence satisfy : s µ ( D ) < 1/2 where µ ( D ) = max m 6=m′ |DtmDm′ | ( 2 ) The sparse approximation Dα1 is a non-linear filtering which preserves the components of β which are `` coherent '' in the dictionary D , represented by few large amplitude coefficients . It eliminates the `` noise '' corresponding to incoherent components of β whose correlations with all dictionary vectors Dm are typically below λ∗ , which can be interpreted as a threshold . Supervised dictionary learning with a deep neural network Dictionary learning for classification amounts to optimizing the matrix D and the threshold λ∗ to minimize the classification loss on a training set { ( xi , yi ) } i . This is a much more difficult non-convex optimization problem than the convex sparse coding problem ( 1 ) . The sparse code α1 of each scattering representation β = LSx depends upon D and λ∗ . It is used as an input to a classifier parametrized by Θ . The classification loss ∑ i Loss ( D , λ∗ , Θ , xi , yi ) thus depends upon the dictionary D and λ∗ ( through α 1 ) , and on the classification parameters Θ . The dictionary D is learned by minimizing the classification loss . This task-driven dictionary learning strategy was introduced in Mairal et al . ( 2011 ) . An implementation of the task-driven dictionary learning strategy with deep neural networks has been proposed in ( Papyan et al. , 2017 ; Sulam et al. , 2018 ; Mahdizadehaghdam et al. , 2019 ; Sun et al. , 2018 ) . The deep network is designed to approximate the sparse code by unrolling a fixed number N of iterations of an iterative soft thresholding algorithm . The network takes β as input and is parametrized by the dictionary D and the Lagrange multiplier λ∗ , as shown in Figure 2 . The classification loss is then minimized with stochastic gradient descent on the classifier parameters and on D and λ∗ . The number of layers in the network is equal to the number N of iterations used to approximate the sparse code . During training , the forward pass approximates the sparse code with respect to the current dictionary , and the backward pass updates the dictionary through a stochastic gradient descent step . For computational efficiency the main issue is to approximate α1 with as few layers as possible and hence find an iterative algorithm which converges quickly . Next section shows that this can be done with homotopy algorithms , that can have an exponential convergence .

Summary: The paper proposes a network architecture composed of three interpretable components followed by a simple MLP classifier. It first applies a scattering transform followed by a learned linear projection (to reduce dimensionality). A sparse representation of these coefficients is then obtained using dictionary learning. The projection, dictionary and MLP classifier are jointly trained to minimize the classification loss. Results show that the model outperforms AlexNet on the Imagenet benchmark.