Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors explicitly design geometrical structure into a CNN by combining it with a Scattering network. This aids stability and limited-data performance. The paper is well written, the contribution of combining Scattering and CNNs is novel and the results seem promising. I feel that such work was a missing piece in the Scattering literature to make it useful for practical applications.

I wish the authors would have investigated the effect of the stable bottom layers with respect to adversarial examples. This can be done in a relatively straightforward way with software like cleverhans [1] or deep fool [2]. It would be very interesting if the first layer's stability in the hybrid architectures increases robustness significantly, as this would tell us that these fooling images are related to low-level geometry. Finding that this is not the case, would be very interesting as well.

Further, the proposed architecture is not evaluated on real limited data problems. This would further strengthen the improved generalization claim. However, I admit that the Cifar-100 / Cifar-10 difference already seems like a promising indicator in this regard.

If one of the two points above will be addressed in an additional experiment, I would be happy to raise my score from 6 to 7.

Summary: 

+ An interesting approach is presented that might be useful for real-world limited data scenarios.
+ Limited data results look promising.
- Adversarial examples are not investigated in the experimental section.
- No realistic small-data problem is addressed.

Minor:
- The authors should add a SOTA ResNet to Table 3, as NiN is indeed out of fashion these days.
- Some typos: tacke, developping, learni.

[1] https://arxiv.org/abs/1610.00768v3
[2] https://arxiv.org/abs/1511.04599

review_title: Interesting work with promising first results.

final_score: rating: null
review_content: Thank you for an interesting read!

In 3.2.2 you replace the Scatter network with a learned CNN, is there a particular reason you consider a 5 layer CNN here, why is it not the same architecture as the fixed one but with learned weights instead?
review_title: 3.2.2 Replacement

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper investigates a hybrid network consisting of a scattering network followed by a convolutional network. By using scattering layers, the number of parameters is reduced, and the first layers are guaranteed to be stable to deformations. Experiments show that the hybrid network achieves reasonable performance, and outperforms the network-in-network architecture in the small-data regime.

I have often heard researchers ask why it is necessary to re-learn low level features of convolutional networks every time they are trained. In theory, using fixed features could save parameters and training time. As far as I am aware, this paper is the first to investigate this question. In my view, the results show that using scattering features in the bottom layers does not work as well as learned CNN features. This is not completely obvious a priori, and so the results are interesting, but I disagree with the framing that the hybrid network is superior in terms of generalization.

For the low-data regime, the hybrid network sometimes gives better accuracy than NiN, but this is quite an old architecture and its capacity has not been tuned to the dataset size. For the full dataset, the hybrid network is clearly outperformed by fully learned models. If I understood correctly, the authors have not simply compared identical architectures with and without scattering as the first layers, which further complicates drawing a conclusion.

The authors claim the hybrid network has the theoretical advantage of stability. However, only the first layers of a hybrid network will be stable, while the learned ones can still create instability. Furthermore, if potentially unstable deep networks outperform stable scattering nets and partially-stable hybrid nets, we have to question the importance of stability as defined in the theory of scattering networks. 

In conclusion, I think the paper investigates a relevant question, but I am not convinced that the hybrid network really generalizes better than standard deep nets. Faster computation (at test time) could be useful e.g. in low power and mobile devices, but this aspect is not really fleshed out in the paper.

Minor comments:
-section 3.1.2: “learni”

review_title: Review

final_score: rating: null
review_content: The paper claims that "A scattering network belongs to the class of convolutional networks whose filters are predefined as wavelets". Is this really true? The scattering network involves convolutions and non-linearities, but unlike a CNN it does not combine information from different channels in one convolution (each channel is convolved separately). It seems like this is a big difference in architecture, aside from the (non)-learnability of the filters.

On a related note, if I remember correctly the wavelet transform as implemented in a computer produces feature maps of different spatial sizes. A CNN expects a uniform size for the input feature maps. How is this dealt with? Do you only use the smallest resolution maps (i.e. the "top layer" of the scattering network)?
review_title: Scattering as a CNN, and downsampling

final_score: rating: 7: Good paper, accept
review_content: Thanks a lot for your detailed response and clarifications.

The paper proposes to use a scattering transform as the lower layers of a deep network. This fixed representation enjoys good geometric properties (local invariance to deformations) and can be thought as a form of regularization or prior. The top layers of the network are trained to perform a given supervised task. This is the final model can be thought as plugging a standard deep convolutional network on top of the scattering transform. Evaluation on CIFAR 10 and 100 shows that the proposed approach achieves performance competitive with high performing baselines.

I find the paper very interesting. The idea of cascading these representations seems very natural thing to try. To the best of my knowledge this is the first work that combines predefined and generic representations with modern CNN architectures achieving competitive performance to high performing approaches. While the state of the art (Resnets and variants) achieves significantly higher performances, I believe that this work strongly delivers it's point.

The paper convincingly shows that lower level invariances can be obtained from analytic representations (scattering transform), simplifying the training process (using less parameters) and allowing for faster evaluation. The of the hybrid approach become crucial in the low data regime. 

The author argues that with the scattering initialisation instabilities cannot occur in the first layers contrary as the operator is non-expansive. This naturally suggests that the model is more robust to adversarial examples. It would be extremely interesting to present an empirical evaluation of this task. What's the practical impact? Can this hybrid network be fooled with adversarial? If this is the case, it would render the use of scattering initialization very attractive.


review_title: review

final_score: rating: null
review_content: 
The paper is very well written and it is nice to read. The idea makes a lot of sense.
In this first stage, a couple of comments and a question. 

I think that it would be interesting to also compare to spatial transformer networks. As the paper shares the same goals with a very different approach. 

In Section 3.1.3, when dealing with small datasets, comparisons against a NiN are presented. Is this network of the same size as the one used with the full dataset? 

There is a typo at the end of this paragraph, the difference with the semi-supervised GAN is larger for 1000 samples. This comparison is a bit unfair with the scattering-CNN. Maybe a good baseline here would be to pre-train a standard network on a different dataset and just fine tune the top layers 
review_title: Baseline

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. Reviewers are generally excited about the combination of predefined representations with CNN architectures, allowing the model to generalize better in the low data regime. This was an extremely borderline paper, and the PCs have determined that the paper would have needed to be further revised and should be rejected.
Decision: Reject