Venue: ICLR 2018 conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: Summary

This article considers neural networks over time-series, defined as a succession of convolutions and fully-connected layers with Leaky ReLU activations. The authors provide relatively general conditions for transformations described by such networks to admit a Lipschitz-continuous inverse. They extend these results to the case where the first layer is a convolution with irregular sampling. Finally, they show that the first convolutional filters can be chosen so as to represent a discrete wavelet transform, and provide some numerical experiments.


Main remarks

While the introduction seemed promising, and I enjoyed the writing style, I was disappointed with this article.

(1) There are many mistakes in the mathematical statements. First, in Theorem 1.1, I do not think that phi_L \circ ... \circ phi_1 \circ F is a non-linear frame, because I do not see why it should be of the form of Definition 1.2 (what would be the functions psi_n?). For the same reason, I also do not understand Theorem 1.2. In Proof 1.4, the line of equalities after « Also with the Plancherel formula » is, in my opinion, not true, because the L^2 norm of a product of functions is not the product of the L^2 norms of the functions. It also seems to me that Theorem 1.3, from [Benedetto, 1992], is incorrect: it is not the limit of t_n/n that must be larger than 2R, but the limit of N_n/n (with N_n the number of t_i's that belong to the interval [-n;n]), and there must probably be a compatibility condition between (t_n)_n and R_1, not only between (t_n)_n and R. In Proposition 1.6, I think that the equality should be a strict inequality. Additionally, I do not say that Proof 2.1 is not true, but the fact that the undersampling by a factor 2 does not prevent the operator from being a frame should be justified.

(2) The authors do not justify, in the introduction, why admitting a continuous inverse should be a crucial criterion of quality for the representation described by a neural network. Additionally, the existence of this continous inverse relies on the fact that the non-linearity that is used is a Leaky ReLU, which looks a bit like "cheating" to me, because the Lipschitz constant of the inverse of a Leaky ReLU, although finite, is large, so it seems to me that cascading several layers with Leaky ReLUs could encode a transformation with strictly positive, but still very poor frame bounds.

(3) I also do not understand why having "orthogonal outputs", as in Section 2, is really desirable; I think that it should be better justified. Also, there are probably other ways to achieve orthogonality than using wavelets in the first layer, so the fact that wavelets achieve orthogonality does not really justify why using wavelets in the first layer is a good choice, compared to other filters.

(4) I had understood in the introduction that the authors would explain how to define a (good) deep representation for data of the form (x_n)_{n\in\N}, where each x_n would be the value of a time series at instant t_n, with the t_n non-uniformly spaced. But all the representations considered in the article seem to be applicable to functions in L^2(\R) only (like in Theorem 1.4 and Theorem 2.2), and not to sequences (x_n)_{n\in\N}. There is something that I did not get here.


Minor remarks

- Fourth paragraph, third line: "this generalization frames"?
- Last paragraph before "Contributions & Organization": "that that".
- Paragraph about notations: it seems to me that what is defined as l^2(R) is denoted as l^2(Z) after the introduction.
- Last line of this paragraph: R^d_1 should be R^{d_1}, and R^d_2 R^{d_2}.
- I think "smooth" could be replaced by "continuous" (smoothness implies a notion of differentiability).
- Paragraph before Proposition 1.1: \sqrt{s} is not defined, and "is supported" should be "are supported".
- Theorem 1.1: the f_k should be phi_k.
- Definition 1.4: "piece-linear" -> "piecewise linear"?
- Lemma 1.2 and Proof 1.4: there are indices missing to \tilde h and \tilde g.
- Proof 1.4: "and finally" -> "And finally".
- Proof 1.5: I do not understand the grammatical structure of the second sentence.
- Proposition 1.4: the definition of a RNN is the same as definition 1.2 (except for the frame bounds); I do not see why such transformations should model RNNs.
- Paragraph before Proposition 1.5: "in,formation".
- Proposition 1.6: it should be said on which space the frame is injective.
- On page 8, "Lipschitz" is erroneously written (twice).
- Proposition 1.7: "ProjW,l"?
- Definition 2.1: in the "nested" property, I think that the inclusion should be the other way around.
- Before Theorem 2.1, the sentence "Such Riesz basis is proven" is unclear to me.
- Theorem 2.1: "filters convolution filters".
- I think the architecture described in Theorem 2.2 could be clarified; I am not exactly sure where all the arrows start from.
- First line of Subsection 2.3: ". is always" -> "is always".
- First paragraph of Subsection 3.2: "the the".
- Paragraph 3.2: could the previous algorithms developed for this dataset be described in slightly more detail? I also do not understand the meaning of "must solely leverage the temporal structure".
- I think that the section about numerical experiments could be slightly rewritten, so that the architecture used in each experiment is clearer. In Paragraph 3.2 in particular, I did not get why the architecture presented in Figure 6 has far fewer parameters than the one in Figure 5; it would help if the authors clearly precised how many parameters each layer contains.
- Conclusion: "we can to" -> "we can".
- Definition 4.1: p_v(s) -> p_v(t).
review_title: There are mistakes in the mathematical statements, and the motivations behind the work are not very clear. I do not think that this article is ready for publication.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
- combination of wavelets & CNN

Cons:
- lack of motivation

I am not sure to understand the motivation of good reconstruction/homeomorphism w.r.t. the numerical setting or combination with a CNN. (except for the first experiment) ; I give some comments section per section

Section 1:
Definition 1.1: why is it squared?
Definition 1.3: with this definition, "the framing constants" are not unique, so it should be "some framing constants"
There is a critical assumption to have an inverse, which is its stability. In particular, the ratio B/A is the quantity of interest. Indeed, a Gaussian filtering is Bi-Lipschitz-Invertible with this definition, yet, however it is quite hard to obtain the inverse which is not stable (and this is the reason why regularization is required in this inverse problem) Consequently, the assumption that CNNs are full rank does not really help in this setting(you can check the singular values). The conditioning is the good quantity to consider.

The Proposition 1.4 is trivial to prove, however I do not understand the following: 
"With such vanishing gradients, it is possible to find series of inputsequences that diverge in l2(Z) while their outputs through the RNN are a Cauchy sequence" 

How would you prove it or do you have some numerical experiments to do so?

Section 2:
The figure of the Theorem 2 is not really clear and could be improved. Furthermore, the 1x1 convolutions which follows the conjugate mirror filtering are not necessarily unitary.. This would require some additional constraints.

Subsection2.3: 
The modulus is missing in the first sentence (on the fourier transform)

Section 3:
I find great the first experiment (which seems to indicate this particular problem is well conditioned). Nevertheless, the second experiment claims to improve the accuracy of the task while reducing the parameters, however it would be great to understand why there is this improvement. Similarly the last problem is better handled by the haar basis, is it because it permits the NN to learns to denoise or is it a conditioning issue? My guess is that it is because this basis sparsify the input signal, but it would require some additional experiments, in particular to understand how the NN uses it.
review_title: Motivation

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors proved that convolutional neural networks with Leaky ReLU activation function are nonlinear frames, and similar results hold for non-uniformly sampled time-series as well. My main concern on this part is that theory is too rough and its link to the later part of the paper is weak. Although frames are stable representations, the ones with lower bound much smaller than the upper bound are close to unstable. That's why in classical applications of frames in signal and image processing tight frames are vastly preferred. Furthermore, the authors did not explicitly state the reliance of the lower frame bound on the parameter alpha in Leaky ReLU. It seems to me that the representation gets more unstable as alpha decreases, and the lower bound will be zero when ReLU is used. 

In Section 2, the authors used CMF conditions to constraint filters which leads to a much more stable representation than in the previous section. The idea is very similar to previous work on data-driven tight frame (Cai et al. Applied & Computational Harmonic Analysis, vol. 37, no. 1, p. 89-105, 2014) and AdaFrame (Tai and E, arXiv:1507.04835). Furthermore, there has been multiple work on introducing tight-frame-like constraints to filters of convolutional neural networks (see for example Huang et al., arXiv:1710.02338). All of these work is not mentioned by the authors. Although the CMF constraints used by the authors seem new, the overall novelty is still weak in my opinion.

The experimental results are convincing, and the proposed architecture with wavelet-transform LSTM outperform the baseline model using standard LSTM. However, I am not familiar with the state-of-the-art models on the data sets used in the paper. Therefore, I am not certain whether the proposed method achieves state-of-the-art or not. 
review_title: The paper proves that a class of convolutional neural networks are nonlinear frames. Experiments demonstrate advantage of introducing CMF constraints on the filters in video classification and financial forecasting

Meta-review: The scores were not favorable: 5,5,2. R2 felt the motivation of the paper was inadequate. R3 raised numerous technical points, some of which were addressed in the rebuttal, but not all. R3 continues to have issue with some of the results. The AC agrees with R3's concerns and feels that the paper cannot be accepted in its current form. 
Decision: Reject