Title: Linear Time Complexity Deep Fourier Scattering Network and Extension to Nonlinear Invariants
Authors: Randall Balestriero
Herve Glotin
Venue: ICLR 2017
Keywords: Unsupervised Learning
Applications
Deep learning

Abstract: In this paper we propose a scalable version of a state-of-the-art deterministic time-
invariant feature extraction approach based on consecutive changes of basis and
nonlinearities, namely, the scattering network. The first focus of the paper is to
extend the scattering network to allow the use of higher order nonlinearities as
well as extracting nonlinear and Fourier based statistics leading to the required in-
variants of any inherently structured input. In order to reach fast convolutions and
to leverage the intrinsic structure of wavelets, we derive our complete model in the
Fourier domain. In addition of providing fast computations, we are now able to
exploit sparse matrices due to extremely high sparsity well localized in the Fourier
domain. As a result, we are able to reach a true linear time complexity with in-
puts in the Fourier domain allowing fast and energy efficient solutions to machine
learning tasks. Validation of the features and computational results will be pre-
sented through the use of these invariant coefficients to perform classification on
audio recordings of bird songs captured in multiple different soundscapes. In the
end, the applicability of the presented solutions to deep artificial neural networks
is discussed.

TL;DR: This paper proposes an extension of the Scattering Network in the Fourier domain and with nonlinear invariant computation for fast and scalable unsupervised representations

Review discussion:
I have few questions:
I do not understand the notion of “higher order non-linearity”. In (Waldspurger 2016), I might be wrong, but I understand “higher order non-linearity” as scattering (or deep features) coefficients of order (e.g. the number of intermediary non-linearity) equal to or more than 3. Could you clarify? Why is your operator a “second order” operator?
The non-linearity you use is not non-expansive when the values are in [-1,1].
Even if ||x||<=1, it does not imply that |a|+|b|<=1.
For example, take x2=y2=x1=-y1=1/sqrt(2), then ||x||=1,||y||=1 and |x2|+|y2|=2/sqrt(2)=sqrt(2)>1. It is however non expansive for smaller values. If the goal is to reduce the SNR, could you replace the square modulus by a soft-thresholding, for instance? It seems to verify the sparsity and contraction property as well, while being used a lot in the signal processing litterature.
Besides, the coefficients might be quickly very small, even just after the first layer: did it lead to numerical errors? How did you normalize them? What is the variance of your representation?
With analytic wavelets, one can roughly consider that the modulus non-linearity provides a smooth envelop of the signal, which means that the frequency of the signal will be concentrated around 0: a low-pass filtering will keep more energy. Is it still the case with a squared modulus? How did you select the path of frequencies you will finally compute?(are they still increasing paths?)
In the equation (12), I do not understand why you introduce S_n. Wouldn’t it be simpler to introduce V_n only as X_n * (I-phi)  or to write it an as empirical variance?
In equation (15), I'm slightly confused: it seems you are mentionning a compact support Fourier filters, but you seem to define the support in space.
In the equation (23), isn't it possible to simplify FoF?
For a fair comparison, did you compare the computation time between your implementation and a scattering implementation with modulus?
Besides, I think the complexity should be explicitely computed with the constants: it depends on the number of filters and the number of fft/ifft performed.
Could you explain why you claim that the features are more discriminative?
I do not understand the figure 4, can you explain it?
In 2.2, why is it important to be invariant to permutations?
Thanks.
I do not understand the notion of “higher order non-linearity”. In (Waldspurger 2016), I might be wrong, but I understand “higher order non-linearity” as scattering (or deep features) coefficients of order (e.g. the number of intermediary non-linearity) equal to or more than 3. Could you clarify? Why is your operator a “second order” operator?
->Higher order nonlinearity refers to |x|^2 instead of |x| as it is usually done in the scattering network.
The non-linearity you use is non-expansive only when the values are in [-1,1].
Even if ||x||<=1, it does not imply that |a|+|b|<=1.
For example, take x2=y2=x1=-y1=1/sqrt(2), then ||x||=1,||y||=1 and |x2|+|y2|=2/sqrt(2)=sqrt(2)>1. A correct proof should use the fact |f(x)-f(y)|<=||f’||_\infty ||x-y|| If the goal is to reduce the SNR, could you replace the square modulus by a soft-thresholding, for instance? It seems to verify the sparsity and contraction property as well, while being used a lot in the signal processing litterature.
Besides, the coefficients might be quickly very small, even just after the first layer: did it lead to numerical errors? How did you normalize them? What is the variance of your representation?
->Soft-thresholding is indeed popular for denoising but the need to estimate the noise standard deviation is problematic as well as the assumption that the noise has constant variance over time. Yes, after many layers, all the coefficients will converge to 0 by definition, however in practice and as it was demonstrated in Mallat's paper, only 2 layers are needed in practice. At most 3 when dealing with extremely chaotic signals.
As a result, we did not experienced any numerical errors. Your suggestion of using the infinite norm is correct.
With analytic wavelets, one can roughly consider that the modulus non-linearity provides a smooth envelop of the signal, which means that the frequency of the signal will be concentrated around 0: a low-pass filtering will keep more energy. Is it still the case with a squared modulus? How did you select the path of frequencies you will finally compute?(are they still increasing paths?)
->It is still the case for the proposed nonlinearity since in the Fourier domain is represents an auto-correlation which by definition has an argmax at 0 which is the mean of the signal in Fourier.
As a result,  you will also "re-concentrate" the energy around 0 with this nonlinearity.
What do you mean by path ?
In the equation (12), I do not understand why you introduce S_n. Wouldn’t it be simpler to introduce V_n only as X_n * (I-phi)  or to write it an as empirical variance?
->As is S defined for the scattering where it could be denoted as an empirical mean, this notation is for clarity purpose so that the corresponding definition is highlighted.
In equation (15), I'm slightly confused: it seems you are mentionning a compact support Fourier filters, but you seem to define the support in space.
->Everything was derived in the Fourier domain.
In the equation (23), isn't it possible to simplify FoF?
->It can be simplified, but the whole point is to not simplify it to take advantage of the different supports for each step.
For a fair comparison, did you compare the computation time between your implementation and a scattering implementation with modulus?
Besides, I think the complexity should be explicitely computed with the constants: it depends on the number of filters and the number of fft/ifft performed.
->As mentioned in the paper, only one FFT is needed, namely on the raw input. Once this is done or if the input was given already in the Fourier domain then only local FFT have to be applied (by local we mean on a really small support size). And thus it is dependent on all the J Q and T parameters. We don't see the motivation to derive the exact complexity as opposed to the asymptotic complexity since in all the scattering papers only the asymptotic complexity is derived, as a result no extra comparison could be done.
Could you explain why you claim that the features are more discriminative?
->The features are more discriminative by the addition of the V features. Since they provide new information about the signal, the concatenation of both features can only be equal or better to the S features alone.
I do not understand the figure 4, can you explain it?
->This figure represent the number of nonzeros coefficients in the Fourier domain w.r.t to the input size as well as the sparsity which is simply the ratio of nonzeros coefficients over the total number of coefficients.
In 2.2, why is it important to be invariant to permutations?
->This is simply a property from the total time invariance property. It is highlighted for more general applications and has not a fundamental sense in our case of bioacoustic signals.
Thank you very much for you questions. Please feel free to ask any other one.
Regards
Overview: This work seems very promising, but I believe it should be compared with more baselines, and more precisely described and explained, from a signal processing point of view.
Pros:
New descriptor
Fast implementation
Cons:
a) Lack of rigor
b) Too long accordingly to the content
c) The computational gain of the algorithm is not clear
d) The work is not compared with its most obvious baseline: a scattering transform
I will detail each cons.
a) Section 1:
The author  motivates the use of scattering transform because it defines a contraction of the space that relies on geometric features.
" The nonlinearity used in the scattering network is the complex modulus which is piecewise linear."
A real modulus is piecewise linear. A complex modulus has a shape of bell when interpreting C as R^2. Could you clarify?
\Omega is not introduced.
Could you give a precise reference (page+paper) of this claim: “Higher order nonlinearity refers to |x|^2 instead of |x| as it is usually done in the scattering network.” ?
Section 2:
The motivation of the non-linearity is not clear.
First, this non-linearity might potentially increase a lot the variance of your architecture since it depends on higher moments(up to 4). I think a fair analysis would be to compute numerically the normalized variance (e.g. divided by the averaged l^2 norm), as a sanity check.
Besides, one should prove that the energy is decreasing. It is not possible to argue that this architecture is similar to a scattering transform which has precise mathematical foundations and those results are required, since the setting is different.
Permutation is not a relevant variability.
The notion of sparsity during the whole paper sometimes refers to the number of 0 value, either the l^1 norm. Mathematically, a small value, even 10^-1000 is still a non 0 value.
Did you compute the graph of the figure 4 on the bird dataset? You might use a ratio instead for clarity.
The wavelet that is defined is not a morlet wavelet ( https://en.wikipedia.org/wiki/Morlet_wavelet ). It is close to be a gabor wavelet, and actually it has not a 0 averaging. The measure of the "sparsity of the filters" is extremely unclear, is it the ratio between the support of the filter and its size? A good criteria might be for instance to understand the amount of the energy that has been neglected.
Besides, a filter with compact support has a bad localisation property.
However, this topic is not  reached in the paper.
For instance, Cauchy wavelets are not used in many applications.(However in mathematical proofs, they often are)
In Subsection 3.4 you write that V^(l)(x)=Sum_{j>l} S^(j)(x), but also that you do compute only the 2 first order coefficients because they can be neglected.
Besides, you specifically write that adding the variance coefficients improve the representation, whereas they can be obtained as linear combination of S.
You claim you apply only one FFT, whereas you apply several FFTs.
b) From "One of the great..." to "iof the
input." section 3.1, the text is not clear. The motivation is that a convolution in space is slower that performing a convolution in the Fourier domain. This whole paragraph can be summarised in few sentences.
The section 3.4 is long and corresponds to implementation details. Maybe it could be removed.
c) The table 2 seems to indicate that the generation of the filters is one of the bottleneck of your software. Is this really true?
One of the main claim of the paper is that sparse filters and staying in Fourier domain speed up the computations. Let us compare the computation of the first order scattering coefficients at scale j with this setting. One has to compare the complexity to compute sum |x*psi_j| and sum |x*psi_j|^2
A downsampling is always performed with wavelets, yet it bears approximation. In a Fourier resolution implementation, one adjust the degree of approximation and speed of computation with the over sampling parameters. Assume a FFT of size N costs C*N*log N, then,
Computing \hat x costs in both case C*N*log N.
ST: Then, the signal is multiplied with the fourier transform of the filter, which has a cost of N. In a fourier multi resolution implementation, one periodises the signal by a factor j, such that its size is N/2^j.
Then, the FFT has cost C*N/2^j*log(N/2^j), and modulus has cost say N.
Then, one applied the averaging that has complexity N/2^j.
Here: The signal is multiplied with the fourier transport of the filter that has a support of N/2^j.
Then, you convolve it with itself, that has thanks to the padding and the FFT a cost of C*N/2^(j-1)*log(N/2^(j-1))+N/2^(j-1). And you take the 0 frequency.
I might be wrong, since this it not my work to do those calculus, but if you claim that your implementation is theoretically faster, you need to prove it, since I do not know any papers where scattering transform claims to be fastly implemented. Here, one sees that the difference is not that significant. Please correct me if I did a mistake.
d) It is essential to compare your work with the representation of a scattering transform.
First, in term of speed of computation, with a fair implementation (e.g. not MATLAB) and secondly in term of accuracy on the dataset you did use: it is a natural baseline.
Thank you for your comments and review.
During the whole paper, the sparsity always define the norm of the representation, it goes from the l1 norm to the 0 norm (number of nonzero coefficients) depending on the context, either the qualitative aspect of the representation or the the Fourier domain sparsity leading to sparse computation and storage as it is for sparse matrices : exactly 0 coefficients.
The whole aim of the paper is to introduce a way to perform computations in a large scale setting by deriving the whole model in the Fourier domain, and not to present a "better" scattering transform. The "variance" feature presented as explained in my other answers can mostly be seen as a way to retrieve the remaining energy at any given layer and is interesting by its simplicity to be computed still in the Fourier domain, it is not the milestone of the approach though.
The wavelet presented is indeed not an analytical wavelet since it does not fulfill mathematically the admissibility criteria.
However in practice and as well for you 10^-1000 example, the value of its mean is negligible.
In fact, the whole paper brings a approximation/speed approach where one is willing to sacrifice some exact modeling and computation in order to be able to compute all the transformation of a deep scattering network on large dataset.
In addition, for the 10^-1000 example, this would be effectively 0 to a computer which makes it irrelevant in this context of sparse approximation.
We indeed apply multiple FFT however they are all on a support of much smaller size than the input signal (they are applied only on the support of the wavelet for each wavelet which by definition are extremely small compared to the input signal, making the M log(M) complexity smaller than N the size of the input signal.
Simply by the fact that we do not have to inverse the Fourier transform to compute the |.| nonlinearity and apply the Fourier transform again to convolve at the next layer means that the presented approach is faster.
Also do not forget than in the scattering transform the number of FFT you have to perform for this grows exponentially with each of the added \lambda_l for each layer.
Also, the presented computation gains seem to be more important than you suggest. In the case of large scale problems, the presented gains can help save tremendous days of computations, for only one signal of course all of this is irrelevant.
The variance coefficients can indeed be retrieve through a linear computation of the following S as mentioned in the paper yet it means that you need many layers of the scattering network to encoded all of them.
Whereas if you stop at layer L you can still retrieve the sum of all the next S by computing this feature, which helps to reduce the number of required layer to capture all the signal energy.
You complexity computation is correct. And in fact, our approach has an asymptotic time complexity equivalent to the scattering transform which is N log (N).
However, in practice, the observed gain is much more important and this is mainly due to the fact that you do not just have less computation but more efficient memory access. Whatever operation applied in our setting access contiguous memory blocks whereas the periodization of the signal after element wise multiplication with the wavelet is quite harmful in practice, just as an example.
So we indeed do not reduce the asymptotic complexity yet the practical results are more than encouraging for large scale datasets.
Finally, when looking at the gains layer per layer, the most gain is obtained at the first layers.
Finally, in Table 2 we show that the construction of the filter-bank grows linearly with the signal size which is thus satisfactory especially since (and as for the scattering network) usually it is created only one time and then applied on all the signals.
Please feel free to ask any clarification.
Also what has not been mentioned here has been changed in the new uploaded version of the paper !
Regards
Thank you for your answers. I still think some improvements are required.
As far as I understood, you software run in 3 hours, and a software using a scattering transform should be in 3*16=2 days. Do you think it would be possible to compare the final accuracy for a same classification pipeline?
I was looking for a precise complexity analysis, in the sens that when I do it on my own, my first calculus(I do mean at the first layer) seems to indicate that the complexity of a scattering transform is better(and not only asmptotical). Your implementation on CPU is more efficient that the implementation of the scattering toolbox, probably for the reasons you invoked. Did you consider using GPUs?
Thank you for your questions.
So first of all concerning the complexity I think there is also a parameter missing in your computation namely the number of wavelet per octave Q.
In fact, this determines directly the actual localization of the wavelet in the Fourier domain and the temporal domain and as Q grows the frequency support shrinks considerably.
Also as Q grows the filter-bank grows and thus the number of filter to apply grows.
As a result it is clear that the standard scattering transform complexity grows.
However in our case, as Q grows the needed convolution in the Fourier domain become less and less complex (since the support shrinks) and this for all the wavelets.
As a result I do agree with you that for low Q (probably between 1 and 4) the standard scattering transform will be faster by definition. The problem is that for audio tasks this frequency resolution leads to poor results (as opposed to images where Q=1 is efficient according to Mallat's papers).
In fact, it is often considered for audio tasks to take at least Q=16 which is roughly the same frequency resolution as the human hear with semi tone.
Concerning the GPU implementation, it is not clear to me if it would lead improvements for a few signals.
Of course if one consider only the computation time then the gain will be huge since parallelization of all the used operators is straightforward.
However when also including the time cost of transporting a signal into the GPU memory required before any GPU computation I think the gain would not be clear anymore given the already low computation time required.
However for large dataset, then as the number of signals grows the more it would be interesting since GPU computation can be done while already transporting the next signal into the GPU.
In fact, I will indeed implement the proposed approach in Theano making it transparent from CPU to GPU.
Finally, I am probably not going to be able to present a better result comparison yet you can compare the proposed results with the paper "Fast Chirplet Transform to Enhance CNN Machine Listening" which is applied on the same dataset.
However I can say that the raw scattering led to poorer results as ii has been used in the bird song challenge and was not the winning solution which used ensemble methods yet we showed that our approach led to as good results.
Please feel free to ask any points not clear in my answer !
Regards.
This work proposes 3 improvements to scattering networks: (1) a non-linearity that allows Fourier-domain computation, (2) compact-supported (in the Fourier domain) representations, and (3) computing additional variance features.
The technical contributions seem worthwhile, since #1 and #2 may result in better speed, while #3 may improve accuracy. Unfortunately, they are poorly described and evaluated. If the writing was clear and the evaluation more broad, I would have recommended acceptance since the ideas have merit.
One of the biggest faults of the presentation is that many sentences are overly long and full of unnecessary obfuscating language, e.g. the last paragraph of Section 1 (though unfortunately this permeates the whole paper).
Likewise, most equations are made unnecessarily complicated.
For example, Eq. 5 does not need 4 lines and so many indexes, but just 2:
X_0 = x
X_l = |X_{l-1} * Psi_l|
with the |.| operator being element-wise. Most of the hyperparameter dependencies and indexes are not necessary, as well as the repetition of iterations. The same reasoning can be applied to most Equations 5 to 13.
The argument of cardinality (Eq. 14) does not really help prove that variance is more informative.
In fact, we could just as easily write that the cardinality of S concatenated with any (!) other quantity is >= the cardinality of S.
Another argument from machine learning theory would be better.
The authors should strive to make the arguments in the paper less hyperbolic and better substantiated. The claims about finding invariants of any input (Abstract) and fundamental structures (last paragraph of Section 1.2.1) are not really backed up by any math. How can we have any guarantees about singling out, for example, semantically relevant representations? The learning procedures in machine learning give at least some guarantees, while here the feature building seems a bit more heuristic. This does not take away from the main idea, but this part needs to be better researched.
Thank you for you consideration.
Concerning the equation, they indeed could be 1 line shorter (not 2 since this would lead to margin problems though) yet I think for clarification purposes, the extra line presents the fact the at each level a new \lambda parameter is introduced which would not be clear otherwise.
You are indeed right on the fact that whatever other concatenated feature by definition could lead to better discrimination.
However the main argument actually comes from the fact that the "variance" feature actually contains the aggregated energy of all the scattering coefficients from the current layer till "infinity" this means that when used at the last layer, it retrieves the remaining energy that otherwise would required more depth in the scattering network.
Please feel free to ask any clarification.
Regards
I find the general direction of the work is promising but, in my opinion, the paper has three main drawback.
While the motivation and overall idea seem very reasonable, the derivation is not convincing mathematically. The experiments are limited and the presentation needs significant improvement.  The writing and wording are in general poorly structured to the point that it is sometimes difficult to follow the proposed ideas. The overall organization needs improvement and the connection between sections is not properly established. The paper could be significantly improved by simply re-writing it.
I'm not fully convinced by the motivation for the proposed non-linearity (|c|^2), as described on page 5. The authors argue that  (Waldspurger, 2016) suggests that higher order nonlinearities might be  beneficial for sparsity.
But unless I'm missing something, that work seems  to suggest that in the general case higher order nonlinearities can be neglected. Could you please comment on this?
On the other hand, adding a second order term to the descriptor seems
an interesting direction, as long as stability to small variations is preserved (which should be shown experimentally)
The experimental section is rather limited. The paper would be stronger with a thorough numerical evaluation. The presented results, in my opinion, do not show convincingly a clear advantage of the proposed method over a standard implementation of the scattering transform. In order to show the merits of the proposed approach, it would be really helpful to directly compare running times and compression rates.
Questions:
- Can you show empirically that the proposed higher order nonlinearity
produces sparser representations than the complex modulus?
Other minor issues:
- The proof of Section 2.1, should be preceded by a clear statement in the form of a proposition
- "Hadamart" -> Hadamard
- "Valid set" -> Validation set
- "nonzeros coefficients" -> nonzero coefficients
- Figure 3 is difficult to understand. Please provide more details.
- Figure 5 is supposed to show a comparison to a standard implementation of the Scattering network, but it doesn't seem to be such comparison in that figure. Please explain.
- Please verify the references. The first reference states "MALLAT".
Thank you for your comments and ideas.
The main motivation for the higher order nonlinearity comes from the induced operation inf the Fourier domain which leads to its computation in the Fourier domain very effectively.
Also, with the right input signal normalization, this will collapse all the coefficients to 0 at a quicker rate than in the standard scattering transform meaning that one needs a reduced number of layers to capture all the energy in the scattering coefficients.
All the writing issues have been changed in the new updated version of the paper.
Adding new descriptors as the one proposed is only intended to show that at the final layer, if not all the energy has been captured yet in the scattering coefficients because the signal was very chaotic, then the remaining energy can be encoded in it since by definition it corresponds to the aggregation of all the scattering coefficients from the given layer to "infinity".
Please let me know if anything needs more clarification.
Regards

Meta-review:
role: PC
data: comment: This paper proposes a to use squared modulus nonlinearities within convolutional architectures. Because point-wise squaring can be written as a convolution in the Fourier domain, when doing all the operations in the Fourier this architecture becomes 'dual': convolutions become pointwise operations, and pointwise square-nonlinearities become convolutions. 
 The authors study this architecture in the context of scattering transforms and produce a complexity analysis that exploits the previous property, along with preliminary numerical experiments. 
 
 All reviewers agreed that, while this is an interesting paper with potentially useful outcomes, its exposition and current experimental section are insufficient. The AC agrees with this assessment, and therefore recommends rejection. 
 I agree that the main unanswered question and a 'show-stopper' is the lack of comparisons with its most immediate baseline, scattering using complex modulus, both in terms of accuracy and computational complexity.

Decision: Reject