Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: For the curse of dimensionality problem of CNNs, this paper presents the Gaussian mixture convolution network (GMCN), a method that alleviate the problem of high-dimension data. GMCN is a deep functional network method that represents data as functions. It fits a Gaussian mixture to result of transfer functions, such as RELUs. The effectiveness of this architecture is verified on MNIST and ModelNet10.
main_review: Strength: This paper provides a new solution to handle the high-dimensional problems.

Weakness:

1.   Compared with other methods, such as PointNet and PointNet++, the effectiveness and practicability of this method cannot be sufficiently demonstrated. It is recommended to validate GMCN on more complex datasets and to give a more adequate analysis. 
2.   TreeHEM is used to reduce the number of Gaussians in the mixture, and  $T=2$ works best in terms of computation time, memory consumption, and often fitting error. The determination of hyper-parameter or specific values are based on experience.
3. The evaluation is quite insufficient as this paper claims that it is a general purpose deep learning The results are not very convincing on complex tasks.

Questions:
1.   In Equation 4, there are $N\times M$ terms, each term is $g_i(x)* g_j(x)$, so according to Equation 5, is the sum of each column substituted in the Relu function? In Equation 7, $B_i$ is unknown? $C$ in Equation 1 and $B$ in Equation 2 should be in bold form.
2.   This paper mentions that "The input GM is then processed by four consecutive blocks of Gaussian convolution  and ReLU fitting layers, where each doubles the number of feature channels and halves the number of Gaussians". So is the first layer $4\times 8$, and the  last layer $64\times 128$, with $4$ Gaussians?
summary_of_the_review: The method presented in this paper is new, but more detailed analysis and evaluation are needed to make this work more solid.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new convolution method to process sparse data in a memory-efficient manner. The authors represent input data, intermediate feature maps and convolution filters as mixtures of Gaussians, which allows computing the result of the convolution analytically. Since the number of mixture components would increase quickly from layer to layer and application of ReLU destroys the property of intermediate layers to be mixtures of Gaussians, they re-fit intermediate feature maps and develop a heuristic to reduce the number of mixture components. The authors evaluate their proposed method on MNIST (2d) and ModelNet10 (3D) and show competitive performance compared to classic PointNet/++ methods.
main_review: ## Strengths
+ Novel, innovative approach
+ Tackles an important problem

## Weaknesses
- Somewhat unclear how small the memory footprint is compared to other methods
- Some central elements like dense fitting and reduction step not very well described
- Somewhat unclear what the limiting factors are for scaling to more complex datasets


## Detailed comments on weaknesses

### Memory footprint

A key argument for the method is that it allows working with sparse (e.g. volumetric) data without running into memory issues. However, after reading the paper I don't have a clear idea about the memory footprint. The number of features (8–>64) and the number of Gaussians (128–>8) doesn't seem very large, yet Table 3 reports ~6 GB memory (these numbers are not very useful, since we don't know the batch size). I wonder whether other methods like OctNets, PointNet etc. wouldn't perform better when matching the memory footprint instead of the number of parameters. After all, memory is the limiting factor when dealing with high-D data, not the number of parameters.


### Central elements not described well

While I understand the motivation and goals behind the dense fitting and reduction step, I could not follow the explanations in section 4.1. Since it's a fairly central part of the method, I think this section needs substantial improvement, both in terms of why things are done the way they are (e.g. Eq. 6+7) and how they are done exactly (TreeHEM).

*[Update on this point after discussions] After some clarifications on the method, I am somewhat unsure about the role of potential negative weights in the convolution kernels. It appears that the method effectively does not use them (or, if so, in a very indirect way).*


### What limits applicability to more complex datasets?

It seems like the methods doesn't work (yet) on ModelNet40. I wonder why this is the case. Is the number of Gaussians and/or feature maps that can be used too small? If that's the case, then does the argument about having a memory-efficient method really hold? If that's not the case, then what is the limiting factor?

summary_of_the_review: I am a bit torn about the paper in its current form. On the one hand it proposes a new and potentially very useful approach. On the other hand, the execution leaves some things to be desired. This makes it a bit difficult to properly assess the benefits and limitations of the method compared to other approaches, in particular more recent ones than PointNet/++. If the authors can clarify the issues mentioned above and provide a convincing argument that the method might scale also to more complex use cases, I'm willing to increase my score.

### Update after rebuttal/discussions
I am still a bit torn, since the intuitions behind e.g. the dense fitting step are still not clear and the role of negative weights. Since some points were clarified, I would give the authors the benefit of the doubt, though.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to rethink deep learning for learning functions 
(with functions being gaussian mixtures GMs, with  positive and/or negative weights unlike for PDFs) as opposed to tensors traditionally used for embedding data. 
Filters are learned as GMs since the cross product (convolution) of two GMMs has a closed form solution.
Applying  an activation/transfer function (e.g. ReLu) to a GM does not produce a GM and 
the paper proposes to approximate a GM from the output of the activation function. Such approach allows to define layers with input/output in GM forms.

main_review:  In a similar spirit to  Functional Data Analysis (Ramsay& Silverman) (e.g. with Functional PCA), the paper proposes to learn  functions with neural networks
which is appropriate when data can be well embedded into GMs (e.g. 3d point cloud).
The paper proposes approaches for  GM fitting so that the output of the activation function (ReLu) becomes a GM.
 As the convolution of GMs produces a GMs with more components (Gaussians), reduction techniques are used to limit the number of components in the GMs as these pass through the network.
The resulting proposed approach shows good performance for classification.

 The cross product (convolution) of two GMMs has a closed form solution. (e.g. as already exploited  for   registration  https://doi.org/10.1109/TPAMI.2010.223 and in statistics http://www.jstor.org/stable/1271214 ). 
Equations  could be better presented e.g. in Eq (1) the weight $a$ should not be part of the Gaussian (as it integrates to  1) and Eq. 2 would best be written as a weighted sum. Results Eq (3) and (4) are known so section 3 acts as a reminder/ state of the art.  
Training time are said to not be competitive against state of the art.

Given that  GMs are chosen as functional representation,  why is ReLu chosen as activation? is there any other activation functions that would have transformed a GM into a GM (apart from the identity function) ? 
For simplicity why is the activation function (ReLu) not applied to the means only (i.e. approximating Gaussian by Dirac ) while applying identity to covariances? (i.e. why is activation not applied in the parameter space of the Gaussian ?) 

summary_of_the_review: The paper is interesting and novel as it rethink neural networks  for learning functions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: CNNs have made an undeniably successful/impact on computer vision tasks. Its core operation relies on grids of discrete data samples, where structure representation of data is used (e.g. vector matrices …). This paper tries to disrupt this well established idea used so far, by proposing a Gaussian of mixtures (GMs) for tackling the curse of dimensionality, that is not avoidable with the conventional CNN architectures. The resulting architecture is termed as a deep functional network.
main_review: Strengths 

This is an interesting work, where the main goal is to replace conventional/standard CNN operations by GM. And this, of course, can  open the door to other innovative approaches that substitute the conventional ones.

Weaknesses

Although I find the proposed approach interesting I would expect some more experimental evaluation to other related approaches and other datasets. Nevertheless, this does not substantially hamper the paper. Another issue that I was struggling with, and I think the main bottleneck of the paper, is the reduction step (namely the use of M Gaussians). And perhaps other choices for eq. (6) and (7) that are not discussed (is it possible to have alternatives ?). See my concern in the detailed review. 


summary_of_the_review: 
When describing the TreeHEM approach, for reducing/merging  (or killing some Gaussian components) I found it rather cumbersome as other methodologies can be used to prune the mixture components. Next, I leave several questions for the discussion:

-it is known that the EM suffers from the problem of the initialization, that is, how many components (per node)  should we represent the data. This often leads to a model selection type of problem. So, how can we deal with this issue automatically in the proposed architecture ?

-Also, it is mentioned that reduction of  the number to Gaussians must be performed. Here, it would be welcome to introduce some of the statistical metrics that are also available to perform this task. In this way it could be possible to have more than one approach, from which a comparison could be performed, and thus, providing more insight into how this could be properly tackled.

-It is not clear the mapping between the proposed context and the clustering hierarchy in Vasconcelos & Lippman (1999) work (that is, levels, blocks etc…).

It is mentioned that a value of T = 2 Gaussian is the best choice. However, this is difficult to follow …

-For completeness, please specify the meaning of the operation in (8).

-When performing the pooling, the fitting process is also performed.Here, it is necessary to enlarge the receptive field. To accomplish this,  the reduction of the covariance diagonal must be decreased. The question is, how is it possible to find the best scaling factor for this purpose ? receptive field . Is it possible to act on the covariance uncertainty for this purpose ?

-it is said that the k-means is used to estimate the initial center’s position. What happens if this procedure does not provide the best location for the centroids ?It is well known that this algorithm may have some limitations regarding this issue.

-Since the proposal is based on the EM algorithm, a clear mapping with this should be done. Specifically, to describe explicitly the E-step and M-step
review_title: null

Meta-review: This paper presents a deep learning method that aims to address the curse-of-dimensionality problem of conventional convolutional neural networks (CNNs) by representing data and kernels with unconstrained ‘mixtures’ of Gaussians and exploiting the analytical form of the convolution of multidimensional Gaussian mixtures. Since the number of mixture components rapidly increases from layer to layer (after convolution) and common activation functions such as ReLU do not preserve the Gaussian Mixtures (GM), the paper proposes a fitting stage that fits a GM to the output of the transfer function and uses a heuristic to reduce the number of mixture components. Experiments are presented on MNIST (2d) and ModelNet10 (3D), which show competitive performance compared to other approaches such as classic CNNs, PointNet and PontNet++ methods.

There is somewhat an overall consensus on the novelty of the proposed approach and its potential to pave the way for further research. There were, however, several issues raised by the reviewers in terms of clarity, memory footprint and computational cost that limits the applicability of the method to more complex datasets. While the authors expanded on the dense fitting in their comments and in the revised version of the paper, it still remains unclear the role of the negative weights, as the dense fitting stage seems to constrain all the weights to be positive. In terms of memory footprint, the authors refer to the theoretical footprint and their implementation does not match this. Finally, it is acknowledged by the authors that the computational cost is a limitation that hinders the method from achieving competitive performance in more complex tasks.
Decision: Accept (Poster)