Title: A HYBRID NETWORK: SCATTERING AND CONVNET

Abstract: This paper shows how, by combining prior and supervised representations, one can create architectures that lead to nearly state-of-the-art results on standard benchmarks, which mean they perform as well as a deep network learned from scratch. We use scattering as a generic and fixed initialization of the first layers of a deep network, and learn the remaining layers in a supervised manner. We numerically demonstrate that deep hybrid scattering networks generalize better on small datasets than supervised deep networks. Scattering networks could help current systems to save computation time, while guaranteeing the stability to geometric transformations and noise of the first internal layers. We also show that the learned operators explicitly build invariances to geometrical variabilities, such as local rotation and translation, by analyzing the third layer of our architecture. We demonstrate that it is possible to replace the scattering transform by a standard deep network at the cost of having to learn more parameters and potentially adding instabilities. Finally, we release a new software, ScatWave, using GPUs for fast computations of a scattering network that is integrated in Torch. We evaluate our model on the CIFAR10, CIFAR100 and STL10 datasets.

1. INTRODUCTION:

Deep architectures builds generic and low-dimensional representations that lead to state-of-the-art results on tasks such as classification (He et al., 2015) , games (Silver et al., 2016) , or generative models (Radford et al., 2015) . These architectures are designed as cascades of non-linear modules that are fully learned. This paper addresses several questions: is it necessary to learn each module? Can a scattering networks replace the first layers? What are the potential benefits?

Hybrid architectures composed of a supervised representation learned on top of an unsupervised representation (Philbin et al., 2007) have been progressively abandoned for the end-to-end training approach (LeCun et al., 2010) . Understanding the nature of the cascade of deep operators is difficult (Szegedy et al., 2013) , since they are learned via back-propagation, and not layer-wise. However, the learned features appear to be transferable to other datasets and helpful for classification (Zeiler & Fergus, 2014) , which implies that the learned representations have captured generic properties for image classification tasks.

Scattering representations (Mallat, 2012) are predefined and generic representations which only require the learning of a few hyper parameters. They consist of a cascade of wavelet transforms and modulus nonlinearities are have proven to be successful in classification tasks such as textures (Bruna & Mallat, 2013b; Sifre & Mallat, 2013) , small digits (Bruna & Mallat, 2013b) , sounds (Andén & Mallat, 2014) or complex image datasets with unsupervised representations (Oyallon & Mallat, 2015) . Nevertheless, these representations do not adapt to the specific bias of each dataset and there is a huge performance gap between supervised and unsupervised representations (Oyallon & Mallat, 2015) .

A convnet is typically a cascade of convolutional layers and nonlinearities, followed by a final average pooling or a sequence of fully connected layers. They lead to state of the art results on CIFAR10 and CIFAR100 (Zagoruyko & Komodakis, 2016) . Some related work to ours (Perronnin & Larlus, 2015) proposed to replace the first layers of convolution by a combination of SIFT and Fisher vec-tors, while learning on top of it a cascade of fully connected layers. This is a hybrid representation in the sense that it combines an unsupervised learned representation and a supervised learned MLP. The numerical results they obtained are competitive on ImageNet with the first AlexNet architecture (Krizhevsky et al., 2012) , while saving computations.

Training a state of the art deep network requires a huge amount of labeled data. Several works tried to tackle this difficulty by developing unsupervised algorithm applied to deepnetwork: for instance evaluating on CIFAR10 an unsupervised generative adversarial method (GAN) pretrained on a subset of Imagenet-1K (Radford et al., 2015) . In a setting where few annotated data are available, training a deep network is hard and requires a lot of regularization, yet a semisupervised learning algorithm applied to a GAN can improve even more the accuracy on CIFAR10, as in Salimans et al. (2016) . Yet, if few data are available, such as in medical imaging, training a deep network from scratch is more complicated: one can only use imagenet pre-trained features (Carneiro et al., 2015) .

Section 2 describes our model, which is a cascade of a scattering network and a convnet. We explain how we build our scattering network, describe its stability properties and exhibit our learning pipeline. Section 3 shows that our network provides competitive results on CIFAR10, CIFAR100 and STL10, while having theoretical guarantees for its representations, in both setting with limited data or not. The experiments can be reproduced using ScatWave1 , an implementation of our algorithm in Torch, which we make publicly available. More details about the software are available in the Appendix A.

2. TOWARDS A HYBRID ARCHITECTURE:

We construct an architecture that consists of two blocks: the first is based on the scattering transform and involves no learning; the second is a classical convnet. In this section, we describe these architectures and their properties.

2.1. SCATTERING NETWORK:

A scattering network belongs to the class of convolutional networks whose filters are predefined as wavelets (Oyallon & Mallat, 2015) . The construction of this network has mathematical foundations (Mallat, 2012) , meaning it is well understood, relies on few parameters and is stable, in contrast deep networks. Stability properties are discussed in Subsection 2.1.2 and Apprendix B. Besides, most of the parameters of this representation does not need to be adapted to the bias of the dataset (Oyallon & Mallat, 2015) , making it a suitable generic representation.

2.1.1. A CASCADE OF WAVELETS AND MODULUS:

In this section, we briefly recall the definition of the scattering transform. It is the cascade of wavelet transforms, and modulus nonlinearity which is finally spatially averaged. Since a modulus is nonexpansive, and a wavelet transform is a linear isometry, a scattering transform is also non-expansive. The local averaging of this representation thus builds a local invariance to translation. In this paper, we only consider a second order scattering network, on the group of translations.

Consider a signal x(u), u ∈ R 2 and an integer J ∈ N, which is the spatial scale of our scattering transform. Let φ J be an averaging with a spatial window of scale 2 J . (for example, a Gaussian averaging) Applying a subsampled averaging A J x(u) = x φ J (2 J u) builds an approximate invariant to translations smaller than 2 J , but it also results in a loss of high frequencies that are necessary to discriminate signals. We define S 0 x = A J x as the order 0 scattering.

A solution to avoid this loss is provided by wavelets. A wavelet is an integrable and localized function in the Fourier and space domain, with a 0 average. A family of wavelets is obtained by dilating a complex mother wavelet ψ (for example, a Morlet wavelet) such that ψ j,θ (u) = 1 2 2j ψ(r -θ u 2 j ), where r -θ is the rotation by -θ, and j ≥ 0 is the scale of the wavelet. A given wavelet ψ j,θ has thus its energy concentrated at a scale j, in the angular sector θ. Let K ∈ N be an integer representing the number of angles of our operator. A wavelet transform is the convolution of a signal with the family of wavelets introduced above, with an appropriate downsampling, i.e.

W 1 x(u, θ 1 , j 1 ) = {x ψ j1,θ1 (2 j1 u)} j≤J,θ=2π l L ,1≤l≤L . Observe that j and θ have been discretized: the wavelet is chosen to be selective in angle and localized in Fourier, thus the sampling is chosen such that (θ 1 , j) → W 1 x(u, θ 1 , j 1 ) is regular enough. Besides, the wavelet transform has been spatially oversampled by a factor 1. The wavelet parameters and this discretization were already chosen in (Oyallon & Mallat, 2015) , where this representation is shown to be generic, so we have used the same hyper-parameters. In their case, {A J x, W 1 x} is approximatively an isometry on the set of signals with limited bandwidth, and this implies the energy of the signal is preserved. This operator belongs to the category of multi-resolution analysis operator, each filter being excited by a specific scale and angle, but the output coefficients are not invariant to translation. We can not apply A J to W 1 x since it gives a trivial invariant, namely 0.

We build the first order scattering coefficients. Applying a point-wise modulus to W 1 x, followed by an averaging A J allows us to build an invariant. If the mother wavelet is analytic, then |W 1 x| is more regular (Bernstein et al., 2013) which implies that the support in Fourier of |W 1 x| is more likely to be contained in a lower frequency domain than W 1 x. Thus, A J preserves the energy of |W 1 x|. In this case, it is possible to define S 1 x = A J |W 1 |x, which can also be written as: S 1 x(u, θ 1 , j 1 ) = |x ψ j1,θ1 | φ J (2 J u); this is the order 1 scattering. It is consequently invariant to translation up to 2 J .

Once more, applying a second wavelet transform W 2 = W 1 on each channels permits the recovery of the high-frequency loss due to the averaging applied to the first order, leading to S 2 x = A J |W 2 ||W 1 |, which can also be written as

S 2 x(u, θ 1 , θ 2 , j 1 , j 2 ) = ||x ψ j1,θ1 | ψ j2,θ2 | φ J (2 J u).

We only compute increasing paths, i.e. j 1 < j 2 because non-increasing paths bear no energy (Bruna & Mallat, 2013b ). We do not compute higher order scatterings, because their energy has been shown experimentally not to be meaningful (Bruna & Mallat, 2013b) .

2.1.2. COVARIANCE AND STABILITY OF THE REPRESENTATION:

In this section, we develop mathematical properties that are obtained by wavelets. Covariance with a group of variability permits building a localized invariant via local averaging. The degree of invariance will be decided by a supervised algorithm, in order to be adapted to the bias of the problem of classification. Here, the parameter J corresponds to a trade-off of invariance in translation and discrimination to adjust A J , and it has to be learned from the data. By construction, as a cascade of convolutions, a scattering network is covariant with translations. Let r θ .x x(r -θ u) be a rotated signal by θ. The representation is still covariant with the rotation in the following sense:

S 1 (r θ .x)(u, θ 1 ) = S 1 x(r -θ u, θ 1 -θ) r θ .(S 1 x)(u, θ 1 ) S 2 (r θ .x)(u, θ 1 , θ 2 ) = S 2 x(r -θ u, θ 1 -θ, θ 2 -θ) r θ .(S 2 x)(u, θ 1 , θ 2 )

However, for S 2 x, the natural set of coordinates that gives a rotational invariant for angles is in fact given by: S2

x(u, θ 1 , α) = S 2 x(u, θ 1 , θ 1 + α)

which naturally leads to:

S2 (r θ .x)(u, θ 1 , α) = S2 x(r -θ u, θ 1 -θ, α) r θ .(S 2 x)(u, θ 1 , α)

In fact, this representation is covariant with the action of the roto-translation group, i.e. R 2 SO 2 (Sifre & Mallat, 2013) . In addition, it can be proven that a scattering network linearizes small deformations (Mallat, 2012) , which means that a linear operator can build invariants to a subset of deformations. It means also that a deep network could reduce the perturbations due to this variability. Furthermore, this representation is complete, in the sense that it is possible to reconstruct a signal from its scattering coefficients (Bruna & Mallat, 2013a) .

It is also important to note that a scattering transform is non-expansive, as a cascade of non-expansive operators, e.g.: Sx -Sy ≤ x -y . Thus, this representation is stable to additive noises, which correspond to perturbations studied in Goodfellow et al. (2016) ; Moosavi-Dezfooli et al. (2015) ; Szegedy et al. (2013) ; Goodfellow et al. (2014) . By adding a very small quantity to an image, the classification performed by a deep network can be fooled, e.g. image x is well classified, but one can find 1 such that x + is not, with the two images being visually indistinguishable.

Apprendix B mathematically quantifies the stability of a hybrid network and proves the unstability potentially rises from the cascaded learned deep network. Indeed, Szegedy et al. (2013) reports that the operators of the different layers of a deep network are not contractive. Corrections were added to fix this by training a deep network on the fooling examples (Goodfellow et al., 2014) , but this requires additional computations. With wavelets as an initialization, instabilities cannot occur in the first layers contrary to Szegedy et al. (2013) , since the operator is non-expansive.

2.2. ADDING SUPERVISION TO OUR REPRESENTATION: CASCADING A CONVNET:

This section introduces our hybrid representation, and explain its interest. We first justify why applying a deep network after scattering is natural. Scattering transforms have yielded excellent numerical results on datasets where the variabilities are completely known, such as MNIST or FERET. In these task, we only encounter the problem of sample variance and handling the variance leads to solving the problem. However, in classification tasks on more complex image datasets, such variabilities are only partially known. Applying the scattering transform on datasets like CIFAR or Caltech leads to nearly state-of-the-art results in the unsupervised case (Oyallon & Mallat, 2015) . But there is a huge gap in performance when comparing to supervised representations, which deep networks can fill in.

We now explain why the scattering transform is an appropriate initialization. Recent works (Mallat, 2016; Bruna et al., 2013) have suggested that a deep network could build an approximation of the group of symmetries of a classification task and apply transformations along the orbits of this group.

The objective of a supervised classifier is to reduce the variabilities due to those symmetries. To each layer corresponds an approximated Lie group of symmetry, and this approximation is progressive, in the sense that the dimension of this approximation is increasing with depth. For instance, the linear Lie group of symmetry of an image is the translation group, R 2 . If no non-linearity is applied, it is not possible to discover new linear groups of symmetry for natural images. In the case of a wavelet transform obtained by rotation of a mother wavelet, it is possible to recover a new subgroup of symmetry, the rotation SO 2 , and the group of symmetry at this layer is the roto-translation group: R 2 SO 2 . Discovering the next groups of symmetry is however a difficult task; nonetheless, the roto-translation group seems to be a good initialization for the first layers. In this work, we investigate this hypothesis.

We thus build a standard deep convolutional network on top of the scattering transform. Its architecture is represented in Figure 2 .2. Our network consists of a cascade of 2C convolutions with spatial kernel size 3 × 3. The C first convolutions use K 0 input channels, except for the first layer, while the C next convolutions have K 1 output channels, except for the last layer. The number of inputs of the first layer is equal to the size of the scattering features, whereas the last layer consists in an average pooling followed by a linear projection. We used a ReLU non-linearity, and no non-linear pooling is involved in our architecture. Observe that if the scattering is applied up to a scale J, then the signal is at spatial resolution J, i.e. its sampling is 2 J , allowing faster computation. In the next section, we discuss the value of those parameters.

Input |W 1 | |W 2 | A J conv 3 × 3 × ×K 0 conv 3 × 3 ×K 0 ×K 0 ... C -1× conv 3 × 3 ×K 0 ×K 1 conv 3 × 3 ×K 1 ×K 1 ... C -1× A P Output

Figure 1 : Architecture of our deep network. Observe that no downsampling is performed Notably, the first layer F 1 of this deep convolutional network is structured by its input, the scattering representation. The nature of this operator and the features selected by this supervised algorithm will be discussed in the next sections.

3. EXPERIMENTS:

We compare our algorithm to supervised, unsupervised and semi-supervised deep networks and evaluate them on the CIFAR10, CIFAR100 and STL10 datasets. CIFAR10 and CIFAR100 datasets consist of colored images of size 32 × 32, with 50000 images in the training set, and 10000 in the test set. CIFAR10 and CIFAR100 have respectively 10 and 100 classes. SLT10 dataset consists of colored images of size 96 × 96, with only 5000 labeled images in the training set divided equally in 10 classes, and 8000 images in the test set. The unlabeled images of this dataset were not used during our experiments. The three datasets are whitened as preprocessing, following the standard procedure. Our software, ScatWave, is implemented in Torch and is publicly available online2 .

3.1 EXPERIMENTAL RESULTS

3.1.1. METHODOLOGY:

During all our experiments, we have trained our architecture with SGD with momentum 0.9 to minimize the standard negative cross-entropy. The batch size is 128. We have applied four types of regularization. First, 0.6 dropout is used after each consecutive two layers. Secondly, we have used 10 -4 weight decay. Futhermore, we have used batch normalization techniques which are supposed to lead to a better conditioning of the optimization (Ioffe & Szegedy, 2015) . Finally, we have augmented the dataset by using random cropping and flipping. The initial learning rate is 0.25 and we divide it by two after every 30 epochs. The networks are trained during 300 epochs.

We now depict the selected hyperparameters of our architecture for CIFAR and STL10 datasets respectively, which we have kept fixed for all experiments unless specifically stated otherwise. For the CIFAR datasets, using cross-validation, the parameters of invariance are set to J = 2 and the number of angles used is L = 8 for the scattering transform. In this case, the output of the scattering network is a tensor of size N

2 J × N 2 J × 3(1 + LJ + 1 2 L 2 J(J -1)) = 8 × 8 × 243

, after reshaping. For the deep net architecture, we chose to use C = 10 layers and K 0 ∈ {128, 512}, K 1 = 128 channels. The number of parameters for CIFAR10 is 9 × (243

× K 0 + (C -1)K 2 0 + K 0 K 1 + (C -1)K 2 1 ) + 10 × K 1 )

, which is roughly equal to 1.6M and 12M parameters for K 0 = 128 and K 0 = 512 respectively. For the STL10 dataset, we chosed K 0 = K 1 = 512 and C = 2: the deep network is shallower to compensate the speed loss due to the use of larger images. In the following, the symbol % corresponds to an absolute percentage of accuracy.

3.1.2. NUMERICAL EXPERIMENTS ON THE ENTIRE DATASET:

We report the classification accuracies in Table 3 .1.2 and discuss them below. We compare our architecture with the unsupervised scattering architecture (Oyallon & Mallat, 2015) . The roto-translation scattering is almost identical to scattering, except that it recombines the channels along the rotation axis by applying a wavelet transform along angles, building more complex geometrical invariants. The classifier of this paper is a RBF SVM kernel, which can be interpreted as a two-layer deep neural network. For the sake of simplicity, we have trained on top of our scattering network a 3 layer fully connected network with size 2048, similarly to (Perronnin & Larlus, 2015) . Without data augmentation, the accuracy of the network is respectively 3.7% and 8.9% below the roto-translation scattering on CIFAR10 and CIFAR100. However, applying data augmentation with translation of length less than 2 2 permits recovering this loss in accuracy, resulting in accuracies of 83.0% and 56.7% (we let the network train for 400 epochs here) respectively on CIFAR10 and CIFAR100. One major difference to our approach is that the system in Oyallon & Mallat (2015) uses a large amount of oversampling. This suggests that in order to learn the appropriate features it is necessary to average the small translation displacement; even if the representation is built to be invariant to translation, its construction relies, for fast computation, on an approximative (but justified) downsampling that leads to non-linear aliasing.

Applying a supervised convnet significantly improves the accuracy of the scattering with 3 fully connected layers, and leads to comparable results with other supervised deep representations: 91.4% on CIFAR10 and 69.5% on CIFAR100. We compare our work with the Highway network (Srivastava et al., 2015) , that consists in a deep cascade of 19 linear and non-linear operators, with an extensive A Wide Resnet (Zagoruyko & Komodakis, 2016) outperforms our architecture by 4.8% on CIFAR10 and 12.2% on CIFAR100, but it requires more engineering process to be designed and is deeper. It is important to recall that, contrary to the wide Resnet, there are no instabilities due to geometric transformations (e.g. translations or deformations), and that the two first layers which are responsible for a large fraction of the computation are not learned, resulting in computational savings. Obviously, the scattering layers do not suffer from the vanishing or exploding gradient issues.

3.1.3. NUMERICAL EXPERIMENTS ON A SMALL SUBSET OF THE DATA:

Supervised deep networks trained on small datasets easily overfit, for instance in the case of medical imaging where little data are available. Semisupervised algorithms exhibit good performances (Salimans et al., 2016) , but it requires a large amount of unlabeled data to work. In this subsection, we show the benefit of using scattering in a framework where those data are not available. We demonstrate that scattering does prevent overfitting on small datasets, while keeping the same architecture and training methodology: this saves time to design an architecture.

For this experiment, we draw several random subsets of CIFAR10 for training our network, and used the same splits for each experiments to train a supervised deep network. Namely, we used a Network in Network (NiN) (Lin et al., 2013) , VGG-like Simonyan & Zisserman (2014) , Wide ResNet Zagoruyko & Komodakis (2016) , which perform better than our network on the full dataset, and we use an implementation available online3 . The VGG did not converge in this situation with the given hyperparameters (such as the depth) and for a simple and fair comparison we decided not to adapt them. We however applied a data augmentation to the inputs of the NiN, VGG and ResNet by translation and flipping. Table 3 corresponds to the averaged accuracy over 5 different subsets, with the corresponding standard deviation. We compare the two architectures with a semisupervised model that consists in a GAN (Salimans et al., 2016) , that is trained on all the data of CIFAR10 yet only a fraction is labeled. With 4000 and 8000 labeled samples, a wide ResNet with 40 layers outperforms by at least 3% the supervised and unsupervised methods which indicate this architecture suffer from less overfitting than the others. If less than 2000 samples are available, a hybrid network outperforms all the supervised architectures: the difference of accuracy between the hybrid architecture and the others is progressively favorable to the hybrid network when the number of samples decreases. Nonetheless, a GAN performs better than a translation scattering with 3 fully connected layers; its performances is almost constant equal to 80%, which shows that this algorithm can adapt itself to the bias of the dataset.

In a second experiment, we apply our hybrid architecture on the STL10 dataset, which is a challenging dataset in the limited sample situation since only 500 samples per class are available. The averaged result is reported in Table 4 . A supervised CNN (Swersky et al., 2013) whose hyper parameters have been automatically tuned achieves 70.1% accuracy. Using the unlabeled data improves by at least 5% the accuracy of a CNN. For an Exemplar CNN (Dosovitskiy et al., 2014) or an Unsupervised Discriminative CNN (Huang et al., 2016) , the weights of the CNNs are unsupervisedly learned from patches of images. Those techniques add several hyper parameters and require an additional engineering process. Applying a hybrid network is straightforward and outpasses both the supervised and unsupervised state of the art by 7.3% and 0.6% respectively.

3.2. RETRAINING MODULES OF THE ARCHITECTURE:

In this section, we show the benefit of using a structured representation. For instance, the first layer of a convnet cascaded on top of a scattering network inherits from the structure of the scattering coefficents. In all the following experiments, we use a converged hybrid network according to the procedure detailed in Subsection 3.1.1.

3.2.1. SIMPLIFYING THE THIRD LAYER OF A DEEPNETWORK:

We numerically analyze the nature of the operations performed along angles by the first layer F 1 of our deep network on CIFAR10. Let us define as F 1 x = {F 0 1 x, F 1 1 x, F 2 1 x} the components associated to the order 0,1,2 scattering coefficients respectively. Let 1 ≤ k ≤ K. In this case, F 0 1 is a convolutional operator that depends on the variables (u, k), F 1 1 depends on (u, θ 1 , k), and F 2 1 depends on (u, θ 1 , α, k). We would like to characterize the smoothness of these operators with respect to the variables, because our representation is covariant to rotations.

To this end, we consider the Fourier transform, for each channels k. We define by F 1 1 , F 2 1 the Fourier transform of these operators along the variables θ 1 and (θ 1 , α) respectively. In space, we applied a DCT transform restrained to its support which is similar to a Fourier transform, since the support of the operator is small. Then the operator is expressed in the tensorial Frequency domain. Since those operations are involutive (up to constants and reflexions), one can easily recover the original operator.

Let us demonstrate how to sparsify this operator in its frequency basis. First, it is possible to threshold by the coefficients of the operators in the Fourier domain, i.e. we replaced the operators

F 1 1 , F 2 1 by 1 | F 1 1 |> F 1 1 and 1 | F 2 1 |> F 2 1 .

Before thresholding, all the frequencies were excited. After this operation, approximatively 10% of coefficients are non-zero. We have tested our network without any retraining and observed a negligible loss of accuracy. We proved that this basis permits a sparse approximation of our filters. Furthermore, we decided to keep only the two first frequencies of F 1 along the variable θ 1 , i.e. we replaced the operators

F 1 1 , F 2 1 by 1 |ω θ 1 |≤1 F 1 1 , 1 |ω θ 1 |≤1 F 2 1 .

This results in a loss of accuracy of roughly 5%. We then fix the layer F 1 , and decide to retrain the next layers, keeping the entire training procedure identical again with an initial learning rate of 1. We obtain a classification accuracy which is 3% below the original architecture. This indicates that the network can almost recover discriminative information from averaged coefficients in angle.

The first experiment shows that in a natural basis it is possible to sparsify the operator. The last experiment indicates that most of the operations performed by the first layer of our network are smooth since they are localized in the Fourier space. This is equivalent to first projecting via an angular averaging each scattering coefficient and it suggests that the system autonomously builds an invariant to geometrical variabilities. While this does not prove that nomore geometrical operators are applied in the next layers, but it does show it is possible to obtain a good accuracy (3% below the original result) with a representation after F 1 which is stable to local roto-translation and deformation variabilities, thanks to a roto-translation averaging.

3.2.2. REPLACING THE SCATTERING NETWORK BY A CONVNET:

Many theoretical arguments of deep learning rely on the universal approximation theorem (Cybenko, 1989) . The flexibility of this deep learning frameworks raises the following question: can we approximate the first scattering layers by a deep network?

In order to explore this question, we consider a 5-layer convnet as a candidate to replace our scattering network on CIFAR10. Its architecture is described on Figure 3 .2.2, and it has the same output size as a scattering network. It has two downsampling steps, in order to mimic the behavior of a scattering network. We keep our architecture identical, except that we replace the scattering part by this network. Then we retrain it, keeping the weights of all the other layers constant and equal to the optimal solution found with the scattering in the previous section. Instead of minimizing a loss between the output of a scattering network and this network, we target the best input for the fixed convnet given the classification task.

This architecture can achieve 1% accuracy below the original pipeline, which is convincing. Using a shallower network seems to degrade the performances, but we did not investigate more this question. Besides, the learned network will not have any guarantee of stability properties. 

Input conv 3 × 3 3× ×128 conv 3 × 3 128× ×128 ↓ 2 conv 3 × 3 128× ×128 conv 3 × 3 128× ×256 conv 3 × 3 256× ×256 ↓ 2 ...

4. CONCLUSION:

We proposed a new deep hybrid architecture that involves scattering and convnets and is competitive with existing approaches. We demonstrate its good generalization performances on CIFAR10, CIFAR100, STL10 and subsets of CIFAR10, CIFAR100. We showed that the cascaded convnet learns an invariant to roto-translation, and that it is possible to learn a deep network that mimics the scattering, at the cost to potentially create instabilities. We release also a fast software to compute a scattering transform on GPUs. This is a preliminary work whose results must be extended to ImageNet. This paper was dedicated to incorporate geometry into deep networks, and we will show how they refine their construction of class invariants in a future work. Computing a scattering transform at order 2 requires computing each path ||x ψ p1 | ψ p2 | where p 1 , p 2 are the parameters with increasing scales of the filters. Computing each path can be viewed as a computational tree, where the coefficient of the scattering transform before an averaging are the leaf of the tree, and the internal nodes are the modulus of the intermediary wavelet transform. The way the tree is walked affects the computation time. In ScatNet (Andén et al., 2014) , the traversal is done by first computing each internal node, storing the results of each internal node. In a second steps, the leafs are computed and stored. In terms of memory, this is not optimal since it requires storing the intermediate computations. Instead, we use an affix traversal of the tree. It reduces at its minimal the memory used, and allow the use of GPUs.

ScatWave is a GPU version of the scattering networks in Torch, that is based on our observation above. ScatNetLight is a MATLAB version on CPUs, which uses as much as possible multithreads.

The Table 5 reports the difference in computation time, for identical parameters and output representations (e.g. same sampling, same hyper parameters). The input corresponds to batches of 128 tensor, the two first dimensions being the size of the image, and the third the number of elements in the tensor (e.g. the color in the case of images) . For comparisons, we used a machine with 24 cores and a TiTan GPU. The speed-up is at least of ×15 in all cases, and up to ×70: ScatWave uses the library cuFFT.

APPENDIX B: A NOTE ON THE STABILITY OF A HYBRID DEEP NETWORK:

In this section, we recall the notion of additive stability of a deep network and derive some simple properties that shows that instabilities are due to the cascaded deepnetwork, and we demonstrate bounds to quantify them. Instabilities are due to perturbations of an initial sample that a deepnetwork does not reduce correctly: the modification is visually not significant or does not affect the label of the perturbated sample, yet the network incorrectly classifies the image. We consider a (trained) deep network f , bounded outputs, that are for instance the probabilities output obtained after a sigmoid function. We write label(x) the labels computed by this deepnetwork for an input x. It is possible to define the contraction factor of a deep network f via: One sees that the amplitude of the resulting instabilities depends on the class of f . Furthermore, the inequalities might be strict and the bounds tighter, but there is no reason it does occur. However, it suggests that such instabilities could be removed if additional constraints were added during the training phase of f .

∆(f ) = sup

The deformation transformations are as well a class of instabilities. The cited works (Goodfellow et al., 2016; Moosavi-Dezfooli et al., 2015; Szegedy et al., 2013; Goodfellow et al., 2014) do not consider them, however, since the scattering transform linearizes them, one sees it is possible for a deepnetwork to explicitly build such invariance. Actually, the average along rotation and translation variables observed in Subsection 3.2.1 seems to indicate it is likely to occur.

:

Code can be found here: https://github.com/edouardoyallon/scatwave

https://github.com/edouardoyallon/scatwave

http://torch.ch/blog/2015/07/30/cifar.html

Figures and Tables:

Figure 2: Architecture of our deep network that mimics a scattering network.

label(x) =label(y) f (x) -f (y)x -yObserve that label(x) = label(y) ⇒ x = y. A small value of ∆(f ) indicates a better stability. It is possible to introduce a local version of this definition, that depends on the input sample (Moosavi-Dezfooli et al., 2015). It is consistant with the definitions of Goodfellow et al. (2016); Moosavi-Dezfooli et al. (2015); Szegedy et al. (2013); Goodfellow et al. (2014), in the sens that the numerator is bounded, but the denominator might become arbitrary small. Let us now consider a hybrid network, e.g. for an input x, the output is f (Sx) where Sx ∈ R n is its scattering transform. Let us write S = {Sx, x} the span of a scattering transform, which corresponds to a strict embedding in the standard euclidean space R n , e.g. S R n . Proposition 1. The following bounds stand:∆(f • S) ≤ sup label(x) =label(ỹ) (x,ỹ)∈S 2 f (x) -f (ỹ) x -ỹ ≤ ∆(f )Proof. Let x, y two samples with different estimated labels, then Sx = Sy. In this case,f (Sx) -f (Sy) x -y = f(Sx) -f (Sy) Sx -Sy . Sx -Sy x -y Thanks to the non-expansivity property of a Scattering transform, Sx -Sy x -y ≤ 1 Setting x = Sx, ỹ = Sy and taking the supremum ends the demonstration.

Accuracy of scattering compared to similar architectures on CIFAR10

Table content: <table><tr><td/><td/><td colspan="2">Table 2: Accuracy of scattering compared to</td></tr><tr><td/><td/><td>similar architectures on CIFAR100</td><td/></tr><tr><td>Architecture</td><td>Accuracy</td><td>Architecture</td><td>Accuracy</td></tr><tr><td>Unsupervised architectures</td><td/><td>Unsupervised architectures</td><td/></tr><tr><td>Roto-translation scattering</td><td>82.3</td><td>Roto-translation scattering</td><td>56.8</td></tr><tr><td>Scattering (ours) + 3 FC</td><td/><td>Scattering (ours) + 3 FC</td><td/></tr><tr><td>+no data augmentation</td><td>78.6</td><td>+ no data augmentation</td><td>47.9</td></tr><tr><td>Scattering (ours) + 3 FC</td><td>83.0</td><td>Scattering (ours) + 3 FC</td><td>56.7</td></tr><tr><td>Supervised hybrid architectures</td><td/><td>Supervised hybrid architectures</td><td/></tr><tr><td colspan="2">Scattering (ours) + CNN, K = 128 89.4</td><td colspan="2">Scattering (ours) +CNN, K = 128 64.4</td></tr><tr><td colspan="2">Scattering (ours) + CNN, K = 512 91.4</td><td colspan="2">Scattering (ours) +CNN, K = 512 69.5</td></tr><tr><td>Supervised architectures</td><td/><td>Supervised architectures</td><td/></tr><tr><td>Highway network</td><td>92.4</td><td>Highway network</td><td>67.8</td></tr><tr><td>All-CNN</td><td>92.8</td><td>All-CNN</td><td>66.3</td></tr><tr><td>Wide ResNet</td><td>96.2</td><td>Wide ResNet</td><td>81.7</td></tr></table>

Accuracy of a hybrid scattering in a limited sample situation on CIFAR10 dataset. N.A. and N.C. stands respectively for Not Available and Not Converged.

Table content: <table><tr><td>Architecture</td><td>1000</td><td>2000</td><td>4000</td><td>8000</td><td>50000</td></tr><tr><td>Supervised architecture</td><td/><td/><td/><td/><td/></tr><tr><td colspan="6">Scattering+CNN, K = 512 58.5±1.2 69.4±0.5 76.2±0.4 81.8±0.7 91.4</td></tr><tr><td>NiN</td><td colspan="5">54.8±1.0 65.1±0.7 71.2±3.8 79.0±3.8 91.9</td></tr><tr><td>VGG</td><td>N.C.</td><td>N.C.</td><td>N.C.</td><td>N.C.</td><td>92.5</td></tr><tr><td>Wide ResNet</td><td colspan="5">58.0±1.2 68.9±1.4 79.1±0.4 86.4±0.3 96.4</td></tr><tr><td>Semi-supervised architecture</td><td/><td/><td/><td/><td/></tr><tr><td>GAN</td><td colspan="5">79.2±2.0 80.4±2.1 82.4±2.3 83.3±1.8 N.A.</td></tr></table>

Accuracy of a hybrid scattering on the STL-10 dataset

Table content: <table><tr><td>Architecture</td><td>Accuracy</td></tr><tr><td>Supervised architectures</td><td/></tr><tr><td colspan="2">Scattering+CNN, K = 512 77.4± 0.4</td></tr><tr><td>CNN</td><td>70.1</td></tr><tr><td>Semi-supervised and</td><td/></tr><tr><td>unsupervised architecture</td><td/></tr><tr><td>Exemplar CNN</td><td>75.4±0.3</td></tr><tr><td>Unsup. Discr. CNN</td><td>76.8±0.3</td></tr></table>

Computation time (in seconds) for different input size, one being with MATLAB on CPUs and the other with Torch on GPUs of multi-resolution computations, such as perfomed in a scattering transform, are delicate. The bottleneck of the scattering on CPU was either speed and memory. It is thus necessary to quickly explain our algorithm: we show that by reorganizing the order of the computation of the algorithm, one can speed them up.

Table content: <table><tr><td>Input size</td><td colspan="3">J ScatNetLight (in s) ScatWave (in s)</td></tr><tr><td>32 × 32 × 3 × 128</td><td>2</td><td>2.5</td><td>0.15</td></tr><tr><td>32 × 32 × 3 × 128</td><td>4</td><td>13</td><td>0.49</td></tr><tr><td>32 × 32 × 3 × 128</td><td>5</td><td>38</td><td>1.1</td></tr><tr><td colspan="2">128 × 128 × 3 × 128 2</td><td>16</td><td>1.0</td></tr><tr><td colspan="2">128 × 128 × 3 × 128 4</td><td>52</td><td>2.3</td></tr><tr><td colspan="2">128 × 128 × 3 × 128 5</td><td>120</td><td>3.7</td></tr><tr><td colspan="2">256 × 256 × 3 × 128 2</td><td>160</td><td>2.2</td></tr><tr><td colspan="4">APPENDIX A: IMPLEMENTATION DETAILS OF SCATWAVE</td></tr></table>

Peer review (iclr2017):
In this paper, the authors explicitly design geometrical structure into a CNN by combining it with a Scattering network. This aids stability and limited-data performance. The paper is well written, the contribution of combining Scattering and CNNs is novel and the results seem promising. I feel that such work was a missing piece in the Scattering literature to make it useful for practical applications. I wish the authors would have investigated the effect of the stable bottom layers with respect to adversarial examples. This can be done in a relatively straightforward way with software like cleverhans [1] or deep fool [2]. It would be very interesting if the first layer*s stability in the hybrid architectures increases robustness significantly, as this would tell us that these fooling images are related to low-level geometry. Finding that this is not the case, would be very interesting as well. Further, the proposed architecture is not evaluated on real limited data problems. This would further strengthen the improved generalization claim. However, I admit that the Cifar-100 / Cifar-10 difference already seems like a promising indicator in this regard. If one of the two points above will be addressed in an additional experiment, I would be happy to raise my score from 6 to 7. Summary: + An interesting approach is presented that might be useful for real-world limited data scenarios. + Limited data results look promising. - Adversarial examples are not investigated in the experimental section. - No realistic small-data problem is addressed. Minor: - The authors should add a SOTA ResNet to Table 3, as NiN is indeed out of fashion these days. - Some typos: tacke, developping, learni. [1] https://arxiv.org/abs/1610.00768v3 [2] https://arxiv.org/abs/1511.04599