Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Overview: This work seems very promising, but I believe it should be compared with more baselines, and more precisely described and explained, from a signal processing point of view.

Pros:
New descriptor
Fast implementation

Cons:
a) Lack of rigor
b) Too long accordingly to the content
c) The computational gain of the algorithm is not clear
d) The work is not compared with its most obvious baseline: a scattering transform

I will detail each cons.

a) Section 1:
The author  motivates the use of scattering transform because it defines a contraction of the space that relies on geometric features.
" The nonlinearity used in the scattering network is the complex modulus which is piecewise linear."
A real modulus is piecewise linear. A complex modulus has a shape of bell when interpreting C as R^2. Could you clarify?
\Omega is not introduced.

Could you give a precise reference (page+paper) of this claim: “Higher order nonlinearity refers to |x|^2 instead of |x| as it is usually done in the scattering network.” ?

Section 2:
The motivation of the non-linearity is not clear. First, this non-linearity might potentially increase a lot the variance of your architecture since it depends on higher moments(up to 4). I think a fair analysis would be to compute numerically the normalized variance (e.g. divided by the averaged l^2 norm), as a sanity check. Besides, one should prove that the energy is decreasing. It is not possible to argue that this architecture is similar to a scattering transform which has precise mathematical foundations and those results are required, since the setting is different.

Permutation is not a relevant variability.

The notion of sparsity during the whole paper sometimes refers to the number of 0 value, either the l^1 norm. Mathematically, a small value, even 10^-1000 is still a non 0 value.

Did you compute the graph of the figure 4 on the bird dataset? You might use a ratio instead for clarity. 

The wavelet that is defined is not a morlet wavelet ( https://en.wikipedia.org/wiki/Morlet_wavelet ). It is close to be a gabor wavelet, and actually it has not a 0 averaging. The measure of the "sparsity of the filters" is extremely unclear, is it the ratio between the support of the filter and its size? A good criteria might be for instance to understand the amount of the energy that has been neglected.

Besides, a filter with compact support has a bad localisation property. However, this topic is not  reached in the paper. For instance, Cauchy wavelets are not used in many applications.(However in mathematical proofs, they often are)

In Subsection 3.4 you write that V^(l)(x)=Sum_{j>l} S^(j)(x), but also that you do compute only the 2 first order coefficients because they can be neglected. Besides, you specifically write that adding the variance coefficients improve the representation, whereas they can be obtained as linear combination of S.

You claim you apply only one FFT, whereas you apply several FFTs.

b) From "One of the great..." to "iof the
input." section 3.1, the text is not clear. The motivation is that a convolution in space is slower that performing a convolution in the Fourier domain. This whole paragraph can be summarised in few sentences.
The section 3.4 is long and corresponds to implementation details. Maybe it could be removed.

c) The table 2 seems to indicate that the generation of the filters is one of the bottleneck of your software. Is this really true?

One of the main claim of the paper is that sparse filters and staying in Fourier domain speed up the computations. Let us compare the computation of the first order scattering coefficients at scale j with this setting. One has to compare the complexity to compute sum |x*psi_j| and sum |x*psi_j|^2

A downsampling is always performed with wavelets, yet it bears approximation. In a Fourier resolution implementation, one adjust the degree of approximation and speed of computation with the over sampling parameters. Assume a FFT of size N costs C*N*log N, then,

Computing \hat x costs in both case C*N*log N. 
ST: Then, the signal is multiplied with the fourier transform of the filter, which has a cost of N. In a fourier multi resolution implementation, one periodises the signal by a factor j, such that its size is N/2^j. Then, the FFT has cost C*N/2^j*log(N/2^j), and modulus has cost say N. Then, one applied the averaging that has complexity N/2^j.

Here: The signal is multiplied with the fourier transport of the filter that has a support of N/2^j. Then, you convolve it with itself, that has thanks to the padding and the FFT a cost of C*N/2^(j-1)*log(N/2^(j-1))+N/2^(j-1). And you take the 0 frequency.

I might be wrong, since this it not my work to do those calculus, but if you claim that your implementation is theoretically faster, you need to prove it, since I do not know any papers where scattering transform claims to be fastly implemented. Here, one sees that the difference is not that significant. Please correct me if I did a mistake.

d) It is essential to compare your work with the representation of a scattering transform. First, in term of speed of computation, with a fair implementation (e.g. not MATLAB) and secondly in term of accuracy on the dataset you did use: it is a natural baseline.
review_title: A promising work, yet maybe a bit immature

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes 3 improvements to scattering networks: (1) a non-linearity that allows Fourier-domain computation, (2) compact-supported (in the Fourier domain) representations, and (3) computing additional variance features.

The technical contributions seem worthwhile, since #1 and #2 may result in better speed, while #3 may improve accuracy. Unfortunately, they are poorly described and evaluated. If the writing was clear and the evaluation more broad, I would have recommended acceptance since the ideas have merit.

One of the biggest faults of the presentation is that many sentences are overly long and full of unnecessary obfuscating language, e.g. the last paragraph of Section 1 (though unfortunately this permeates the whole paper).

Likewise, most equations are made unnecessarily complicated. For example, Eq. 5 does not need 4 lines and so many indexes, but just 2:
X_0 = x
X_l = |X_{l-1} * Psi_l|
with the |.| operator being element-wise. Most of the hyperparameter dependencies and indexes are not necessary, as well as the repetition of iterations. The same reasoning can be applied to most Equations 5 to 13.

The argument of cardinality (Eq. 14) does not really help prove that variance is more informative. In fact, we could just as easily write that the cardinality of S concatenated with any (!) other quantity is >= the cardinality of S. Another argument from machine learning theory would be better.

The authors should strive to make the arguments in the paper less hyperbolic and better substantiated. The claims about finding invariants of any input (Abstract) and fundamental structures (last paragraph of Section 1.2.1) are not really backed up by any math. How can we have any guarantees about singling out, for example, semantically relevant representations? The learning procedures in machine learning give at least some guarantees, while here the feature building seems a bit more heuristic. This does not take away from the main idea, but this part needs to be better researched.

review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: I find the general direction of the work is promising but, in my opinion, the paper has three main drawback. While the motivation and overall idea seem very reasonable, the derivation is not convincing mathematically. The experiments are limited and the presentation needs significant improvement.  The writing and wording are in general poorly structured to the point that it is sometimes difficult to follow the proposed ideas. The overall organization needs improvement and the connection between sections is not properly established. The paper could be significantly improved by simply re-writing it.

I'm not fully convinced by the motivation for the proposed non-linearity (|c|^2), as described on page 5. The authors argue that  (Waldspurger, 2016) suggests that higher order nonlinearities might be  beneficial for sparsity. But unless I'm missing something, that work seems  to suggest that in the general case higher order nonlinearities can be neglected. Could you please comment on this?

On the other hand, adding a second order term to the descriptor seems
an interesting direction, as long as stability to small variations is preserved (which should be shown experimentally)

The experimental section is rather limited. The paper would be stronger with a thorough numerical evaluation. The presented results, in my opinion, do not show convincingly a clear advantage of the proposed method over a standard implementation of the scattering transform. In order to show the merits of the proposed approach, it would be really helpful to directly compare running times and compression rates.

Questions: 
- Can you show empirically that the proposed higher order nonlinearity
produces sparser representations than the complex modulus?

Other minor issues:
- The proof of Section 2.1, should be preceded by a clear statement in the form of a proposition
- "Hadamart" -> Hadamard
- "Valid set" -> Validation set
- "nonzeros coefficients" -> nonzero coefficients
- Figure 3 is difficult to understand. Please provide more details.
- Figure 5 is supposed to show a comparison to a standard implementation of the Scattering network, but it doesn't seem to be such comparison in that figure. Please explain.
- Please verify the references. The first reference states "MALLAT".

review_title: Review

final_score: rating: null
review_content: I have few questions:

I do not understand the notion of “higher order non-linearity”. In (Waldspurger 2016), I might be wrong, but I understand “higher order non-linearity” as scattering (or deep features) coefficients of order (e.g. the number of intermediary non-linearity) equal to or more than 3. Could you clarify? Why is your operator a “second order” operator?

The non-linearity you use is not non-expansive when the values are in [-1,1]. Even if ||x||<=1, it does not imply that |a|+|b|<=1. For example, take x2=y2=x1=-y1=1/sqrt(2), then ||x||=1,||y||=1 and |x2|+|y2|=2/sqrt(2)=sqrt(2)>1. It is however non expansive for smaller values. If the goal is to reduce the SNR, could you replace the square modulus by a soft-thresholding, for instance? It seems to verify the sparsity and contraction property as well, while being used a lot in the signal processing litterature. Besides, the coefficients might be quickly very small, even just after the first layer: did it lead to numerical errors? How did you normalize them? What is the variance of your representation?

With analytic wavelets, one can roughly consider that the modulus non-linearity provides a smooth envelop of the signal, which means that the frequency of the signal will be concentrated around 0: a low-pass filtering will keep more energy. Is it still the case with a squared modulus? How did you select the path of frequencies you will finally compute?(are they still increasing paths?)

In the equation (12), I do not understand why you introduce S_n. Wouldn’t it be simpler to introduce V_n only as X_n * (I-phi)  or to write it an as empirical variance?

In equation (15), I'm slightly confused: it seems you are mentionning a compact support Fourier filters, but you seem to define the support in space.

In the equation (23), isn't it possible to simplify FoF?

For a fair comparison, did you compare the computation time between your implementation and a scattering implementation with modulus? Besides, I think the complexity should be explicitely computed with the constants: it depends on the number of filters and the number of fft/ifft performed.

Could you explain why you claim that the features are more discriminative?

I do not understand the figure 4, can you explain it?

In 2.2, why is it important to be invariant to permutations?

Thanks.
review_title: Non-linearity questions

Meta-review: This paper proposes a to use squared modulus nonlinearities within convolutional architectures. Because point-wise squaring can be written as a convolution in the Fourier domain, when doing all the operations in the Fourier this architecture becomes 'dual': convolutions become pointwise operations, and pointwise square-nonlinearities become convolutions. 
 The authors study this architecture in the context of scattering transforms and produce a complexity analysis that exploits the previous property, along with preliminary numerical experiments. 
 
 All reviewers agreed that, while this is an interesting paper with potentially useful outcomes, its exposition and current experimental section are insufficient. The AC agrees with this assessment, and therefore recommends rejection. 
 I agree that the main unanswered question and a 'show-stopper' is the lack of comparisons with its most immediate baseline, scattering using complex modulus, both in terms of accuracy and computational complexity.
Decision: Reject