Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper deals with the problem of learning nonlinear operators using deep learning. Specifically, the authors propose to extend deep neural networks to the case where hidden layers can be infinite-dimensional. They give results on the quality of the approximation using these operator networks, and show how to build neural network layers that are able to take into account topological information from data. Experiments on MNIST using the proposed deep function machines (DFM) are provided. 

The paper attempts to make progress in the region between deep learning and functional data analysis (FDA). This is interesting. Unfortunately, the paper requires significant improvements, both in terms of substance and in terms of presentation. My main concerns are the following:

1) One motivation of DFM is that in many applications data is a discretization of a continuous process and then can be represented by a function. FDA is the research field that formulated the ideas about the statistical data analysis of data samples consisting of continuous functions, where each function is viewed as one sample element. This paper fails to consider properly the work in its FDA context. Operator learning has been already studied in FDA. See for e.g. the problem of functional regression with functional responses. Indeed the functional model considered in the linear case is very similar to Eq. 2.5 or Eq. 3.2. Moreover, extension to nonparametric/nonlinear situations were also studied. The authors should add more information about previous work on this topic so that their results can be understood with respect to previous studies.

2) The computational aspects of DFM are not clear in the paper. From a practical computational perspective, the algorithm will be implemented on a machine which processes on finite representations of data. The paper does not clearly provide information about how the functional nature and the infinite dimensional can be handled in practice. In FDA, generally this is achieved via basis function approximations.

3) Some parts of the paper are hard to read. Sections 3 and 4 are not easy to understand. Maybe adding a section about the notation and developing more the intuition will improve the reading of the manuscript. 

4) The experimental section can be significantly improved. It will be interesting to compare more DFM with its discrete counterpart. Also, other FDA approaches for operator learning should be discussed and compared to the proposed approach.

review_title: This paper proposes a deep neural network approach to learn nonlinear operators. The current version of the paper requires significant improvements, both in terms of substance and in terms of presentation.

final_score: rating: 7: Good paper, accept
review_content: This paper extends the framework of neural networks for finite-dimension to the case of infinite-dimension setting, called deep function machines. This theory seems to be interesting and might have further potential in applications.
review_title: DEEP FUNCTION MACHINES: GENERALIZED NEURAL NETWORKS FOR TOPOLOGICAL LAYER EXPRESSION

final_score: rating: 3: Clear rejection
review_content: The main idea of this paper is to replace the feedforward summation
y = f(W*x + b)
where x,y,b are vectors, W is a matrix
by an integral
\y = f(\int W \x + \b)
where \x,\y,\b are functions, and W is a kernel. A deep neural network with this integral feedforward is called a deep function machine. 

The motivation is along the lines of functional PCA: if the vector x was obtained by discretization of some function \x, then one encounters the curse of dimensionality as one obtains finer and finer discretization. The idea of functional PCA is to view \x as a function is some appropriate Hilbert space, and expands it in some appropriate basis. This way, finer discretization does not increase the dimension of \x (nor its approximation), but rather improves the resolution. 

This paper takes this idea and applies it to deep neural networks. Unfortunately, beyond rather obvious approximation results, the paper does not get major mileage out of this idea. This approach amounts to a change of basis - and therefore the resolution invariance is not surprising. In the experiments, results of this method should be compared not against NNs trained on the data directly, but against NNs trained on dimension reduced version of the data (eg: first fixed number of PCA components). Unfortunately, this was not done. I suspect that in this case, the results would be very similar. 


review_title: a functional analysis view of neural networks but without a major theorem

Meta-review: The idea of extending deep nets to infinite dimensional inputs is interesting but, as the reviewers noted, the execution does not have the quality we can expect from an ICLR publication. I encourage the authors to consider the meaningful comments that were made and modify the paper accordingly.
Decision: Reject