Title: A hybrid network: Scattering and Convnet
Authors: Edouard Oyallon
Venue: ICLR 2017
Keywords: Computer vision
Unsupervised Learning
Deep learning

Abstract: This paper shows how, by combining prior and supervised representations, one can create architectures that lead to nearly state-of-the-art results on standard benchmarks, which mean they perform as well as a deep network learned from scratch. We use scattering as a generic and fixed initialization of the first layers of a deep network, and learn the remaining layers in a supervised manner. We numerically demonstrate that deep hybrid scattering networks generalize better on small datasets than supervised deep networks. Scattering networks could help current systems to save computation time, while guaranteeing the stability  to geometric transformations and noise of the first internal layers. We also show that the learned operators explicitly build invariances to geometrical variabilities, such as local rotation and translation, by analyzing the third layer of our architecture.  We demonstrate that it is possible to replace the scattering transform by a standard deep network at the cost of having to learn more parameters and potentially adding instabilities. Finally, we release a new software, ScatWave, using GPUs for fast computations of a scattering network that is integrated in Torch. We evaluate our model on the CIFAR10, CIFAR100 and STL10 datasets.

TL;DR: This paper shows how, by combining prior and supervised representations, one can create architectures that lead to nearly state-of-the-art results on standard benchmarks.

Review discussion:
The paper claims that "A scattering network belongs to the class of convolutional networks whose filters are predefined as wavelets". Is this really true? The scattering network involves convolutions and non-linearities, but unlike a CNN it does not combine information from different channels in one convolution (each channel is convolved separately). It seems like this is a big difference in architecture, aside from the (non)-learnability of the filters.
On a related note, if I remember correctly the wavelet transform as implemented in a computer produces feature maps of different spatial sizes. A CNN expects a uniform size for the input feature maps. How is this dealt with? Do you only use the smallest resolution maps (i.e. the "top layer" of the scattering network)?
Dear reviewer,
Thank you for your questions, I answer below to them.
"Comment: The paper claims that "A scattering network belongs to the class of convolutional networks whose filters are predefined as wavelets". Is this really true? The scattering network involves convolutions and non-linearities, but unlike a CNN it does not combine information from different channels in one convolution (each channel is convolved separately). It seems like this is a big difference in architecture, aside from the (non)-learnability of the filters."
Thank you for pointing out that this sentence is not clear.
Indeed, there is also a final spatial averaging after each cascade of wavelets, as described in Section 2.1.1. It could be also added that it is a complex valued network. You are correct, each channel is convolved separately. This is a special case of CNNs.
Indeed, as you said, a CNN can be defined as the cascade of a linear operator covariant with translation (e.g. "different channels in one convolution") and a point-wise non-linearity. As a cascade of wavelet transforms and modulus non-linearity, the linear operators of a scattering network are covariant with translation. In the Subsection 2.1 where the scattering transform is presented, it is written "$W_2=W_1$" which is a slight abuse of nations that you might have noticed.
Another paper ( http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Oyallon_Deep_Roto-Translation_Scattering_2015_CVPR_paper.pdf ) refines the comparison between a CNN and a scattering network, by showing that actually a wavelet transform can be interpreted as a deep cascade of conjugate mirror filters and downsampling (e.g. $h$ and $g_\theta$ with the notations of this paper) whose weights are shared. Those filters are analogs to the convolutionnal kernels of a CNN. I will add this reference here in order to clarify this sentence.
Furthermore, the special variants of the translation scattering transforms such as scattering transform on the roto-translation or separable roto-translation group can recombine the channels across angles by applying a wavelet transform along angles. Those are more sophisticated architectures that were not used here.
"On a related note, if I remember correctly the wavelet transform as implemented in a computer produces feature maps of different spatial sizes. A CNN expects a uniform size for the input feature maps. How is this dealt with? Do you only use the smallest resolution maps (i.e. the "top layer" of the scattering network)?"
As explained in the Subsection 2.1.1 and the end of the Section 2.1, a final averaging is applied at the output of the cascade of the wavelet transform. It will fix the final sampling of the output tensor accordingly to the spatial averaging, e.g. $J$. A scattering networks of order 2 outputs the 0 order ($S_0$), first order ($S_1$) and second order ($S_2$) scattering coefficients, that have the same sampling since they use the same spatial averaging. Because each scattering coefficient has the same spatial resolution, the output can be concatenated into a tensor, which is fed to a CNN.
Consequently, there are 3 dimensions: two indexes for the spatial variable, and the third corresponds to the channels.
Thank you again for your questions and let me know if I should clarify some specific points.
Best,
EO
As mentioned in another comment, the topologies of the scattering network and CNN are different.
In fact, in the scattering network, the number of dimension grows for each layer as you develop the network.
However for a CNN there is an aggregation of the last layer filters so that the shape is always a 3D tensor.
In addition, filters are not multiscale at a given layer as opposed to the scattering network where there is a complete wavelet filter-bank.
So given that, is it really better to use fully deterministic first layers as opposed to for example letting the ANN optimize the used wavelets in a parametric estimation framework for example ?
Thank you.
Regards.
Thank you for your comment.
A CNN does not perform an aggregation, but it performs a linear operation that is covariant with the action of translation, and so do a Scattering Network. I agree that the number of dimension grows, however, the spatial size is also down-sampled at each layer. And the wavelet filter-bank cascade can be formulated as a CNN implementation. All those elements implies that the Scattering Network is a constrained CNN.
The first layers are not fully deterministic, in the sens that the parameters of a Scattering Network are cross-validated, which is a supervised learning algorithm. I totally agree it would be definitely possible to learn  mother wavelets, so that they adapt to the specific biases of a dataset, and this would be an interesting work.
However, I do believe the most complicated part to understand deep networks is to define the directions of variabilities, which could correspond to groups of symmetry. Scattering Transform bears structures that might help to tackle this difficulty, by structuring the internal layers of a deep network. And this work seems to indicate it might also act as a form of regularization by incorporating knowledge of geometric variabilities.
Thank you again for your comment and let me know if I could clarify some points.
Best,
EO
Thank you for an interesting read!
In 3.2.2 you replace the Scatter network with a learned CNN, is there a particular reason you consider a 5 layer CNN here, why is it not the same architecture as the fixed one but with learned weights instead?
Dear reviewer,
Thank you for your comment!
Also, thank you for your question.
First, I tried several architectures, shallower, and I could not achieve a good accuracy. The objective was to prove one can replace the scattering network by a CNN.
However, the fact I could not train networks of depth less than 5 is not a proof that it is not possible, and I will try to give your my intuition on this question.
In the setting of this paper, with this implementation, a scattering transform can be interpreted as a complex deep network with depth 3 (there 3 linear operators and 2 non linearity).
However, the non-linearity is really different. If a filter at the first layer of a real deepnetwork was corresponding to a real wavelet, a ReLU might create undesirable singularity (when the signal oscillates and gets negative) that might lead to artifact. To avoid this kind of artifacts, I think it is necessary to increase the depth so that the next layer can handle this instability (at the potential cost to create new ones..).
Furthermore, the support of the filters of the CNN is chosen equal to 3. It means one can obtain filters (by cascading them and removing ReLUs) with support 3, 5, 7, ... etc.
Yet, a Morlet wavelet has no compact support: it makes the approximation by those filters more complex and requires specific design.
Consequently, one must cascade more than two convolutional layers in this case. And a non-linearity is added because usually, in deep learning litterature, people claim it helps.
Best,
EO
The paper is very well written and it is nice to read. The idea makes a lot of sense.
In this first stage, a couple of comments and a question.
I think that it would be interesting to also compare to spatial transformer networks. As the paper shares the same goals with a very different approach.
In Section 3.1.3, when dealing with small datasets, comparisons against a NiN are presented. Is this network of the same size as the one used with the full dataset?
There is a typo at the end of this paragraph, the difference with the semi-supervised GAN is larger for 1000 samples. This comparison is a bit unfair with the scattering-CNN. Maybe a good baseline here would be to pre-train a standard network on a different dataset and just fine tune the top layers
Dear reviewer,
Thank you for your comment!
Thank you for your suggestion. I agree there is a link with Spatial Transformers. A Spatial Transformer module builds an explicit transformation of the image which is estimated, in order to help to minimize the loss function. My understanding of the mechanism is that this technique implictely builds the orbits of predefined group of variabilities, and then selects a specific element along this orbit, which helps for the classification task. In some sens, it is an invariant, yet not a representation.
However, in the case of a scattering network, we build a representation invariant locally along the orbit of variability, while being discriminative, thanks to wavelets that permits to recover the loss frequencies. And this is very different.
Besides, the Spatial Transformer modules seem to be applied on datasets such as rotated & distorted MNIST, SVHN or GRSRB, and to remove variabilities that correspond to global elastic distorsion, global affine transformation.
However, those variabilities might not be present in the CIFAR datasets. (I do not know any papers applying this technique on CIFAR10, yet I would be glad if you had some references) Besides, it requires the learning of more parameters than a scattering transform, it does not build linear invariants to variabilities and it does not correspond to an interpretable representation of signals.
Yet, I agree a baseline based on this technique would be interesting to study.
I answer to your questions below.
"In Section 3.1.3, when dealing with small datasets, comparisons against a NiN are presented. Is this network of the same size as the one used with the full dataset? "
Yes. And the same network was used for the Hybrid Scattering CNN. It is important to keep the architectures identical for a fair comparison. I believe it is possible to find a CNN architecture that would lead to a better accuracy than the hybrid network, by reducing for instance the number of parameters.
However, it would have required more engineering effort and the obtained network would not probably get a good accuracy when using the full dataset.
Besides, I believe that incorporating the knowledge of roto-translation acts as a regularization whose network can benifits in a limited sample situation. This situation occurs for instance in medical imaging.
"There is a typo at the end of this paragraph, the difference with the semi-supervised GAN is larger for 1000 samples. This comparison is a bit unfair with the scattering-CNN. Maybe a good baseline here would be to pre-train a standard network on a different dataset and just fine tune the top layers "
You are mentionning the sentence: "For 1000 samples, there is a difference of 4.3% but only a difference of 2:9%with 8000samples." The absolute percentage corresponds to the difference between the supervised NiN and Hybrid network, I will clarify this.
In the case of the semi supervised GAN, observe that we are not in a limited sample situation.
Indeed, only 1 000 labels are known, but the remaning 49 000 samples are all used to optimize the architecture. I will clarify this. In the case of the NiN, VGG (that does not converge) or Hybrid network, we only use 1000 samples for training, the architecture being identical to the case of 50 000 samples. The  goal is not to compare the network with a pretrained architecture, that might incorporate refined invariants since during the optimization it might have accessed to more samples, but it is to observe the degradation of the numerical accuracies in limited samples situations.
Thank you again for your questions and comments, and let met know if I should clarify some points.
Best,
EO
Thanks a lot for your detailed response and clarifications.
The paper proposes to use a scattering transform as the lower layers of a deep network. This fixed representation enjoys good geometric properties (local invariance to deformations) and can be thought as a form of regularization or prior. The top layers of the network are trained to perform a given supervised task. This is the final model can be thought as plugging a standard deep convolutional network on top of the scattering transform. Evaluation on CIFAR 10 and 100 shows that the proposed approach achieves performance competitive with high performing baselines.
I find the paper very interesting. The idea of cascading these representations seems very natural thing to try. To the best of my knowledge this is the first work that combines predefined and generic representations with modern CNN architectures achieving competitive performance to high performing approaches.
While the state of the art (Resnets and variants) achieves significantly higher performances, I believe that this work strongly delivers it's point.
The paper convincingly shows that lower level invariances can be obtained from analytic representations (scattering transform), simplifying the training process (using less parameters) and allowing for faster evaluation. The of the hybrid approach become crucial in the low data regime.
The author argues that with the scattering initialisation instabilities cannot occur in the first layers contrary as the operator is non-expansive. This naturally suggests that the model is more robust to adversarial examples. It would be extremely interesting to present an empirical evaluation of this task. What's the practical impact? Can this hybrid network be fooled with adversarial? If this is the case, it would render the use of scattering initialization very attractive.
Dear reviewer,
I thank you for your positive review.
I complete my global answer specifically to your review, for which I thank you again. Each deepnet seems to be fooled by small valued vectors (https://arxiv.org/pdf/1610.08401.pdf ), and if no additional constraints are added during the optimization process, there is no reason that the hybrid network becomes more robust. In the Appendix B, I have added a note that shows that the amplitude of the instabilities of the hybrid network is likely to be of the order of magnitude of the instabilities of the cascaded network. Bounds are derived.
However, a scattering network is stable to other source of transformation such as deformation, as developed in the paper, which means a scattering network is potentially a good initialization.
Thank you very much for your remarks, comments that were helpful to resubmit this paper and thank you again for your time.
Best regards,
EO
The paper investigates a hybrid network consisting of a scattering network followed by a convolutional network. By using scattering layers, the number of parameters is reduced, and the first layers are guaranteed to be stable to deformations. Experiments show that the hybrid network achieves reasonable performance, and outperforms the network-in-network architecture in the small-data regime.
I have often heard researchers ask why it is necessary to re-learn low level features of convolutional networks every time they are trained.
In theory, using fixed features could save parameters and training time. As far as I am aware, this paper is the first to investigate this question. In my view, the results show that using scattering features in the bottom layers does not work as well as learned CNN features. This is not completely obvious a priori, and so the results are interesting, but I disagree with the framing that the hybrid network is superior in terms of generalization.
For the low-data regime, the hybrid network sometimes gives better accuracy than NiN, but this is quite an old architecture and its capacity has not been tuned to the dataset size. For the full dataset, the hybrid network is clearly outperformed by fully learned models. If I understood correctly, the authors have not simply compared identical architectures with and without scattering as the first layers, which further complicates drawing a conclusion.
The authors claim the hybrid network has the theoretical advantage of stability.
However, only the first layers of a hybrid network will be stable, while the learned ones can still create instability.
Furthermore, if potentially unstable deep networks outperform stable scattering nets and partially-stable hybrid nets, we have to question the importance of stability as defined in the theory of scattering networks.
In conclusion, I think the paper investigates a relevant question, but I am not convinced that the hybrid network really generalizes better than standard deep nets. Faster computation (at test time) could be useful e.g. in low power and mobile devices, but this aspect is not really fleshed out in the paper.
Minor comments:
-section 3.1.2: “learni”
Dear reviewer,
I thank you for your constructive review that was very helpful for the (hopefully) final submission of this work.
Thank you again for your comments and several issues you highlight. I would like to answer to each of them.
The paper does not claim that the hybrid network is strictly superior in term of generalization on the full dataset, and I do not believe this would be possible, with such a wide litterature of incremental improvements obtained via a huge process of hyper parameters fine tuning. (Or at least until the mathematical fundations are fully understood).
Instead, we fix a simple architecture that was not fine tuned, since this is simply a preliminary work. The author has indeed positively investigates the question of cascading a resnet, but this is not the purpose of this paper.
However, the comparison with a resnet in the limited sample situation is very interesting. It shows that the resnet architecture is really robust to limited sample situation on CIFAR datasets, but it is beaten by the hybrid architecture when really few samples are available, e.g. 2000 samples. Adapting the architecture of those deep networks to the limited sample situation (e.g. reducing the number of parameters w.r.t. the number of samples) will add another process of selecting hyper parameters that is desirable to avoid. In order to convince you that scattering networks are an enjoyable initialization, I have added a state-of-the-art result on STL10 that leads to 77.4% accuracy. STL10 is a challenging dataset with large images, 10 classes and 500 samples per class.
I believe the notion of stability is not specific to the scattering litterature, yet it has mathematical foundations. Concretely talking, let us take the example of a navigation system of a car which is controlled by a deep network: for the safety of its passengers, such system should never be fooled by small additive perturbations. I believe this is a central topic to build robust systems and that it is extremely connected to understanding deep networks.
I agree that the additive stability of the whole network is unlikely to happen, and this is proven in the Appendix B of this paper. It shows the instabilities of the hybrid network are bounded by the instabilities of the learned network. The resulting instabilities come from the cascaded and learned deep network. Such instabilities could be avoided if they were correctly handled during the optimization process, yet more work in this direction has to be adressed, and I believe a scattering transform is the first step to this kind of work, because this property is explicitly implemented at no cost.
However, as developed in  the section 2.1.2, the stability to other sources (such as deformation) of variability is explicitly obtained. Unfortunately, I do not know a software that permits to quantify these and this is not the purpose of this paper
Finally, I do agree that faster computation time are one of the natural outcome of this paper.
While being not explicitely adressed here, in our future work, this will be a central question not only for mobile device issues but as well for processing bigger dataset such as imagenet.
I thank you again for your comments, review and time that were very helpful to improve this paper.
Best regards,
EO
In this paper, the authors explicitly design geometrical structure into a CNN by combining it with a Scattering network. This aids stability and limited-data performance. The paper is well written, the contribution of combining Scattering and CNNs is novel and the results seem promising. I feel that such work was a missing piece in the Scattering literature to make it useful for practical applications.
I wish the authors would have investigated the effect of the stable bottom layers with respect to adversarial examples. This can be done in a relatively straightforward way with software like cleverhans [1] or deep fool [2]. It would be very interesting if the first layer's stability in the hybrid architectures increases robustness significantly, as this would tell us that these fooling images are related to low-level geometry. Finding that this is not the case, would be very interesting as well.
Further, the proposed architecture is not evaluated on real limited data problems. This would further strengthen the improved generalization claim.
However, I admit that the Cifar-100 / Cifar-10 difference already seems like a promising indicator in this regard.
If one of the two points above will be addressed in an additional experiment, I would be happy to raise my score from 6 to 7.
Summary:
+ An interesting approach is presented that might be useful for real-world limited data scenarios.
+ Limited data results look promising.
- Adversarial examples are not investigated in the experimental section.
- No realistic small-data problem is addressed.
Minor:
- The authors should add a SOTA ResNet to Table 3, as NiN is indeed out of fashion these days.
- Some typos: tacke, developping, learni.
[1] https://arxiv.org/abs/1610.00768v3
[2] https://arxiv.org/abs/1511.04599
Dear reviewer,
I thank you for your positive and constructive review.
I have added the results of the Wide Resnet in the Table 3, as well as the VGG, in the limited sample situation setting. They interestingly show  that a resnet is  extremly robust to these situations. I should add, but I will not put it in the final paper since this is not the purpose of this work, that cascading a resnet on top of a scattering network does not reduce (when the appropriate J parameter is selected) the resulting accuracy on the whole dataset.
I have added a SOTA result on STL10, which leads to 77.4% accuracy. This dataset is a challenging dataset with few labeled samples available.  I agree this was a missing result, and I thank you for this suggestion.
However, I believe CIFAR datasets are also interesting since one can observe the evolution of the accuracy with respect to the amount of available labeled data. Interestingly, a scattering network that does not lead to SOTA beats the resnet when 2000 samples are available.
I thank you for your observation about the stability w.r.t. additive perturbations. Unfortunately, I did not use the softwares you mention since the implementations are unfortunately not done in Lua, however I believe this will not be a limitation.
Indeed, in the Appendix B of this paper, I did theoritically quantify  the instabilities. One observes that the scattering is unlikely to reduce the additive instabilities that are due to the learned and cascaded deep network ; recent works seem to indicate those instabilities are always present( https://arxiv.org/pdf/1610.08401.pdf ).
However a scattering transform can help to build invariance to other source of instabilities such as deformations.
I thank you again very much for your helpful comments, questions, review and your time.
Best regards,
EO
Thank you for your response, additional results and Appendix B, very interesting.
I have updated my rating.
Dear reviewers,
Here are two elements that I have added thanks to your constructive and helpful suggestions:
- I have added a note in the Appendix B that quantifies precisely the additive perturbations of a Deep network. It proves the unstability of the hybrid deepnetwork is always smaller than the unstability of the cascaded deepnetwork and discusses the equality case.
Besides, no straightforward softwares were available in Lua. Since I am convinced that without an appropriate constraints during the optimization the deep network will not perform a contraction, I decided not to investigate experimentally this question.
However a scattering transform can build invariance to deformations for instance, which is not raised (to my knowledge) by the works that try to fool deepnetworks.
- I have added as well state-of-the-art results on the STL10 dataset that leads to  77.4% accuracy. I would like to highlight that no specific fine tuning of the architecture was performed, and that using the scattering layers is quite straightforward.
I will put in few weeks the code online on github, since it needs to be cleaned. For the potential readers of this review that might be interested in this work, we will release in few weeks a software that we scaled for Imagenet, which will be available on my website.
Thank you again very much for your remarks and comments that were helpful to improve this paper.
Best regards,
EO

Meta-review:
role: PC
data: comment: The program committee appreciates the authors' response to concerns raised in the reviews. Reviewers are generally excited about the combination of predefined representations with CNN architectures, allowing the model to generalize better in the low data regime. This was an extremely borderline paper, and the PCs have determined that the paper would have needed to be further revised and should be rejected.

Decision: Reject