Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces the notion of continuous convolutional neural networks. 
The main idea of the paper is to project examples into an RK Hilbert space
and performs convolution and filtering into that space. Interestingly, the
filters defined in the Hilbert space  have parameters that are learnable.

While the idea may be novel and interesting, its motivation is not clear for
me. Is it for space? for speed? for expressivity of hypothesis spaces? 
Most data that are available for learning are in discrete forms and hopefully,
they have been digitalized according to Shannon theory. This means that they bring
all necessary information for rebuilding their continuous counterpart. Hence, it is
not clear why projecting them back into continuous functions is of interest. 

Another point that is not clear or at least misleading is the so-called Hilbert Maps.
As far as I understand, Equation (4) is not an embedding into an Hilbert space but
is more a proximity space representation [1]. Hence, the learning framework of the
authors can be casted more as a learning with similarity function than learning
into a RKHS [2]. A proper embedding would have mapped $x$ into a function
belonging to $\mH$. In addition, it seems that all computations are done
into a \ell^2 space instead of in the RKHS (equations 5 and 11). 
Learning good similarity functions is also not novel [3] and Equations
(6) and (7) corresponds to learning these similarity functions.
As far as I remember, there exists also some paper from the nineties that
learn the parameters of RBF networks but unfortunately I have not been able to
google some of them.


Part 3 is the most interesting part of the paper, however it would have been
great if the authors provide other kernel functions with closed-form convolution 
formula that may be relevant for learning.
The proposed methodology is evaluated on some standard benchmarks in vision. While
results are pretty good, it is not clear how the various cluster sets have been obtained
and what are their influence on the performances (if they are randomly initialized, it 
would be great to see standard deviation of performances with respect to initializations).
I would also be great to have intuitions on why a single continuous filter works betters
than 20 discrete ones (if this behaviour is consistent accross initialization).

On the overall, while the idea may be of interested, the paper lacks in motivations
in connecting to relevant previous works and in providing insights on why it works.
However, performance results seem to be competitive and that's the reader may
be eager for insights.


minor comments
---------------

* the paper employs vocabulary that is not common in ML. eg. I am not sure what
occupancy values, or inducing points are. 

* Supposingly that the authors properly consider computation in RKHS, then \Sigma_i
should be definite positive right? how update in (7) is guaranteed to be DP? 
This constraints may not be necessary if instead they used proximity space representation.





[1] https://alex.smola.org/papers/1999/GraHerSchSmo99.pdf
[2] https://www.cs.cmu.edu/~avrim/Papers/similarity-bbs.pdf
[3] A. Bellet, A. Habrard and M. Sebban. Similarity Learning for Provably Accurate Sparse Linear Classification. 
review_title: nice idea. lack in motivation and insights

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to provide a continuous variant of CNN. The main idea is to apply CNN on Hilbert maps of the data. The data is mapped to a continuous Hilbert space via a reproducing kernel and a convolution layer is defined using the kernel matrix. A convolutional Hilbert layer algorithm is introduced and evaluated on image classification data sets.

The paper is well written and provides some new insights on incorporating kernels in CNN.

The kernel matrix in Eq. 5 is not symmetric and the kernel function in Eq. 3 is not defined over a pair of inputs. In this case, the projections of the data via the kernel are not necessarily in a RKHS. The connection between Hilbert maps and RKHS in that sense is not clear in the paper.

The size of a kernel matrix depends on the sample size. In large scale situations, working with the kernel matrix can be computational expensive. It is not clear how this issue is addressed in this paper.

In section 2.2, how \mu_i and \sigma_i are computed?

How the proposed approach can be compared to convolutional kernel networks (NIPS paper) of Mairal et al. (2014)?
review_title: This paper proposes an extension of CNN to the case where data features are continuous functions in a RKHS. The paper is well written and provides some new insights on incorporating kernels in CNN.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper formulates a variant of convolutional neural networks which models both activations and filters as continuous functions composed from kernel bases. A closed-form representation for convolution of such functions is used to compute in a manner than maintains continuous representations, without making discrete approximations as in standard CNNs.

The proposed continuous convolutional neural networks (CCNNs) project input data into a RKHS with a Gaussian kernel function evaluated at a set of inducing points; the parameters defining the inducing points are optimized via backprop. Filters in convolutional layers are represented in a similar manner, yielding a closed-form expression for convolution between input and filters. Experiments train CCNNs on several standard small-scale image classification datasets: MNIST, CIFAR-10, STL-10, and SVHN.

While the idea is interesting and might be a good alternative to standard CNNs, the paper falls short in terms of providing experimental validation that would demonstrate the latter point. It unfortunately only experiments with CCNN architectures with a small number (eg 3) layers. They do well on MNIST, but MNIST performance is hardly informative as many supervised techniques achieve near perfect results. The CIFAR-10, STL-10, and SVHN results are disappointing. CCNNs do not outperform the prior CNN results listed in Table 2,3,4. Moreover, these tables do not even cite more recent higher-performing CNNs. See results table in (*) for CIFAR-10 and SVHN results on recent ResNet and DenseNet CNN designs which far outperform the methods listed in this paper.

The problem appears to be that CCNNs are not tested in a regime competitive with the state-of-the-art CNNs on the datasets used. Why not? To be competitive, deeper CCNNs would likely need to be trained. I would like to see results for CCNNs with many layers (eg 16+ layers) rather than just 3 layers. Do such CCNNs achieve performance compatible with ResNet/DenseNet on CIFAR or SVHN? Given that CIFAR and SVHN are relatively small datasets, training and testing larger networks on them should not be computationally prohibitive.

In addition, for such experiments, a clear report of parameters and FLOPs for each network should be included in the results table. This would assist in understanding tradeoffs in the design space.

Additional questions:

What is the receptive field of the CCNNs vs those of the standard CNNs to which they are compared? If the CCNNs have effectively larger receptive field, does this create a cost in FLOPs compared to standard CNNs?

For CCNNs, why does the CCAE initialization appear to be essential to achieving high performance on CIFAR-10 and SVHN? Standard CNNs, trained on supervised image classification tasks do not appear to be dependent on initialization schemes that do unsupervised pre-training. Such dependence for CCNNs appears to be a weakness in comparison.
review_title: review: experimental results not compelling

Meta-review: The paper received borderline negative scores: 5,6,4.

The authors response to R1 question about the motivations was "...thus can achieve similar classification results with much smaller network sizes. This translates into smaller memory requirements, faster computational speeds and higher expressivity." If this is really the case, then some experimental comparison to compression methods (e.g. Song Han's PhD work at Stanford) is needed to back up this.

R4 raises issues with the experimental evaluation and the AC agrees with them that they are disappointing. In general R4 makes some good suggestions for improving the paper.

The author's rebuttal also makes the general point that the paper should be accepted as it contains ideas, that these are sufficient alone: "We strongly believe that with some fine-tuning it could achieve considerably better results, however we also believe that this is not the point in a first submission...". The AC disagrees with this. Ideas are cheap. *Good ideas*, i.e. those that work, as in get good performance on standard benchmarks are valuable however. The reason for having benchmarks is to give some of objective way of seeing if an idea has any merit to it. So while the reviewers and the AC accept that the paper has some interesting ideas, this is not enough for warrant acceptance. 
Decision: Reject