Gaussian Mixture Convolution Networks
Authors: Adam Celarek
Pedro Hermosilla
Bernhard Kerbl
Timo Ropinski
Michael Wimmer
Venue: ICLR 2022 Poster (2022)
Keywords: deep learning architecture
gaussian convolution
gaussian mixture
3d

Abstract: This paper proposes a novel method for deep learning based on the analytical convolution of multidimensional Gaussian mixtures.
In contrast to tensors, these do not suffer from the curse of dimensionality and allow for a compact representation, as data is only stored where details exist.
Convolution kernels and data are Gaussian mixtures with unconstrained weights, positions, and covariance matrices.
Similar to discrete convolutional networks, each convolution step produces several feature channels, represented by independent Gaussian mixtures.
Since traditional transfer functions like ReLUs do not produce Gaussian mixtures, we propose using a fitting of these functions instead.
This fitting step also acts as a pooling layer if the number of Gaussian components is reduced appropriately.
We demonstrate that networks based on this architecture reach competitive accuracy on Gaussian mixtures fitted to the MNIST and ModelNet data sets.

Published as a conference paper at ICLR 2022
GAUSSIAN MIXTURE CONVOLUTION NETWORKS
Adam Celarek Pedro Hermosilla', Bernhard Kerbli, Timo Ropinski', Michael Wimmer! 'TU Wien
Ulm University
ABSTRACT
This paper proposes a novel method for deep learning based on the analytical convolution of multidimensional Gaussian mixtures. In contrast to tensors, these do not suffer from the curse of dimensionality and allow for compact representation, as data is only stored where details exist. Convolution kernels and data are Gaussian mixtures with unconstrained weights, positions, and covariance matrices. Similar to discrete convolutional networks, each convolution step produces several feature channels, represented by independent Gaussian mixtures. Since traditional transfer functions like ReLUs do not produce Gaussian mixtures, we propose using fitting of these functions instead. This fitting step also acts as a pooling layer if the number of Gaussian components is reduced appropriately. We demonstrate that networks based on this architecture reach competitive accuracy on Gaussian mixtures fitted to the MNIST and ModelNet data sets.
INTRODUCTION
Convolutional neural networks (CNNs) have led to the widespread adoption of deep learning in many disciplines. They operate on grids of discrete data samples, i.c., vectors, matrices, and tensors, which are structured representations of data. While this is approach is sufficiently efficient one and two dimensions, it suffers the curse of dimensionality: The amount of data is O(d"), ), where d is the resolution of data, and i is the dimensionality. The exponential growth permeates all layers of CNN, which must be scaled appropriately to learn sufficient number of k-dimensional features. In three or more dimensions, the implied memory requirements thus quickly become intractable (Maturana & Scherer, 2015; Zhirong Wu al., 2015), leading rescarchers to propose specialized CNN architectures as a trade-off between mathematical exactness and performance (Riegler et al., 2017; Wang et al., 2017). In this work, propose novel deep learning architecture based on the analytical convolution of Gaussian mixtures (GMs). While maintaining the elegance of conventional CNNs, our architecture does not suffer from the curse of dimensionality and is therefore well-suited for application to higher-dimensionall problems.
GMs are inherently unstructured and sparse, in the sense that storage is required to represent empty data regions. In contrast to discretized k-dimensional volumes, this allows for a more compact representation of data across dimensions, preventing exponential memory requirements. In this regard, they are similar to point clouds (Qi et al., 2017a). However, GMs also encode the notion of a spatial extent. The fidelity of the representation directly depends on the number of Gaussians in the mixture, which means that, given an adequate fitting algorithm and sufficient resources, it can be arbitrarily adjusted. Hence, GMs allow trading representation accuracy for memory requirements on more fine-grained level than voxel grids.
In this work, we present our novel architecture, the Gaussian mixture convolation network (GMCN), as alternative to conventional CNNs. An important distinguishing feature of GMCNs is that both, data and Icarnable kernels, are represented using mixtures of Gaussian functions. We use Gaussians with unconstrained weights, positions, and covariances, enabling the mixtures to adapt more accurately to the shape of data than a grid of discrete samples could. Unlike probability models, approach supports negative weights, which means that arbitrary data can be fitted.
At their core, CNNs are deep neural networks, where each pixel or voxel is a neuron with distinct connections to other neurons, with shared weights and biases. On the other hand, GMCNs represent

---

Published as a conference paper at ICLR 2022
data as functions. Hence there are distinct neurons, and the concept of connection does not exist. Our architecture could thus be more appropriately classified as a deep fuecttonn network.
Like conventional (discrete) CNNs, our proposed GM learning approach supports several convolution layers, an increasing number of feature channels, and pooling, in order to learn and detect aspects of the input. For the convolution of two mixtures, a closed-form solution is used, which results in new mixture. Conventional transfer functions like ReLUs do not produce a GM. However, a GM is required to feed the next convolution layer in a deep network. Therefore, we propose to fit GM to the result of the transfer function. The fitting process can simultaneously act as pooling layer if the number of fitted components is reduced appropriately in successive convolution layers. Unlike most point convolutional networks, which have per-point feature vectors, the feature channels of GMCNs are not restricted to share the same positions or covariance matrices. Compared to conventional CNNs, GMCNs exhibit very compact theoretical memory footprint in 3 and more dimensions (derivation and examples are included in Appendix A of this paper).
With the presentation of GMCNs, we thus provide the following scientific contributions:
A deep learning architecture based on the analytical convolution of Gaussian mixtures.
A heuristic for fitting a Gaussian mixture to the result of a transfer function.
A fitting method to both reduce the number of Gaussians and act as a pooling layer.
A thorough evaluation of GMCNs, including ablation studies.
2 RELATED WORK
CNNs have been successfully applied to images (c.g.. Krizhevsky et al. (2012)) and small 3D voxel grids (c.g., Maturana & Scherer (2015), and Ji et al. (2013)). However, is not trivial to apply discrete convolution large 3- more dimensional problems due to the dimensional explosion. Riegler et al. (2017) reduce memory consumption by storing data in an octree, but the underlying problem remains.
GMs have been used in 3D computer graphics, e.g., by Jakob al. (2011) for approximating volumetric data, and by Preiner et al. (2014) and Eckart et al. (2016a) for a probabilistic model of point clouds. In particular, the work by Eckart et al. can be used to fit GMs to point clouds, which can then be used as input to our network (see ablation study in Section 5.3). We use the cnnttationmmaximmzzatiin (EM) algorithm proposed by Dempster et al. (1977) to fit input data, and EM equations for fitting small GMs to larger ones by Vasconcelos & Lippman (1999).
We refer to Ahmed et al. (2018) for a general overview of deep learning on 3D data. Like point clouds, GMs are unordered but with additional attributes (weight and covariance). These additional attributes could be processed like feature vectors attached to points. Therefore, prior work on point clouds is relevant, whereby Guo et al. (2019) provide good overview.
Several authors propose point convolutional networks, e.g., Li et al. (2018b), Xu et al. (2018), Hermosilla et al. (2018), Thomas et al. (2019), Boulch (2019), and Atzmon et al. (2018). The one by Atzmon et al. is most similar to ours. They add a Gaussian with uniform variance for every point, creating mixture that is coarse fitting of the point cloud. After analytical convolution, the result is sampled at the original locations of the point cloud. However, information away from these points is lost. Effectively, such restricted convolution creates weighting function for neighboring points.
Radial Basis Function Networks also learn a GM to detect patterns on the input data (Broomhead & Lowe, 1988). However, the goal of such networks differs from our approach. While their goal is, similar to MIPs, to detect patterns on a set of data points, our method uses radial basis functions to detect patterns using convolution operator on another set of radial basis functions.
GAUSSIAN MIXTURES AND CONVOLUTION IN MULTIPLE DIMENSIONS
multidimensional Gaussian function is defined as
g(x, b, C) = ((2T)) det (C) e {a (m--b) (mm-b)
(I)
2

---

Published as a conference paper at ICLR 2022
channels in
repeat for all Fo output channels
ReLU
fit
*
F
P. x No.
F x No.
number of Geussians
Ng
Figure I: Design of Gaussian convolution layer: The inputs F; channels of GMs with N; components each, which analytically convolved with A; learnable kernels, Nr components each. The convolved mixture contains F; x N₂ components. After applying ReLU, a new GM with Ny components fitted. This repeated for all the F. output feature channels. The resulting tensor becomes the input of the next convolution channel, with the number of output channels Fo becoming F,, and N, becoming N₁. The fitting must he fully differentiable with respect to the input of the ReL.U. In this example, F; 8, N; 16, N₂- 5, and N₂ 8.
where & the number of dimensions, the shift center point, and C' the shape (covariance matrix) of the Gaussian. A GM is a weighted sum of multiple such Gaussians:
gm( a, B,C)
Σ ai g(x B,, C.) 1=0
(2)
GM a probabilistic model since the weights a; can be negative and do not necessarily sum up to 1. The convolution of two Gaussians according to Vinga (2004) is given by
g(x, b, C) g(r, B) = g (ar, B,C +T.
(3)
Due to the distributive property of convolution, we have:
N
M
M
Σ 7==
(x)
(x)
Σ
8m (2) = Σ
en +gm (x)) m=0m=0
(
(4)
m==
where g; (x) is a short hand for a g(x B, C.). Clearly, the convolution of one mixture with N terms with another mixture with M terms yields mixture with N x M terms.
4 GAUSSIAN MIXTURE CONVOLUTION NETWORKS
Conventional CNNs typically consist of several feature channels, convolution layers, a transfer func- tion, and pooling layers. this section, we will show that all these components can implemented by GMCNs well. As result, overall architecture is similar to discrete CNNs, except that data and kermels are represented by Gaussian functions rather than pixels or voxels.
Convolution layer. All batch elements and feature channels contain the number of compo- nents, allowing for packing Gaussian data (i.c., weights, positions, and covariances) into a single tensor. Like in conventional CNNs, each output channel (F)) in GMCN the sum of all F; input channels, convolved with individual kemels (Figure 1). In practice, this means that multiple convolu- tion results are concatenated in the convolution layer. This results in a mixture with N, F,N,N components, where N, and N₂ are component counts in input and kernel mixtures, respectively.
Transfer function. The ReLU is commonly used transfer function, whose ramp-like shape is defined by plz) max(0, (0, x). When applying this transfer function to mixture gm, we get
((gm(x)) max (0.
8n (x))
We can use Equation evaluate the mixture at particular locations of æ, as shown in Figure 2b. However, any subsequent convolution layers in GMCN will again require GM input. Another
3

---

Published as a conference paper at ICLR 2022
(a) Input
(b) Target
(c) Initialization
(d) Dense fitting (e) Reduction slep
Fitting after 1 convolution
+1
160 Gaussians
16 Gaussians
2nd convolation
Firting after
-1
640 Gaussians
8 Gaussians
Figure Examples of litting pipeline described Section 4.1: The input mixture shown (a) (b) after applying the ReLU, (c) and (d) show the coarse approximation (a') and final weights (a" of the dense fitting. The mixture after the reduction step shown in (c). The overall fitting error is bound by dense fitting, as theorctically, the reduction can be omitted altogether (or very high number of Gaussians used).
problem inherent to the definition of GM convolution in Section 3, is that the number of intermediate output components (and thus implied memory requirements hidden layers) is set to increase with each convolution. We address both of these problems by introducing dedicated fitting step that produces new, smaller mixture to represent the result (ggm(x)) and act as input for subsequent GMCN layers.
4.1 GAUSSIAN MIXTURE FITTING
Fitting a GM to the output of the transfer function is not trivial. In order to propagate the gradient the kernels, the fitting method must be differentiable with respect the parameters of the Gaussian components in gm. A naive option is to sample the GM, evaluate ip, and perform an EM fitting. However, this quickly becomes prohibitive due the costs associated with sampling several points from each Gaussian, iterations of likelihood computations, and computation of the associated gradients. We decided first fit dense GM the transfer function output (e.g., a ReLU (x)), shown in Figure 2b), and then reduce the number of Gaussians in a separate step.
Dense fitting. reduce the degrees of freedom that we must optimize for during fitting, we first assume that some Gaussians in the output of the last convolution step are meaningfully placed. Hence, we freeze their positions and adjust only their weights. One possible solution to adjust the GM's weights would be to set up overdetermined linear system, defined by set of sample points in the mixture and the corresponding output values. This set of equations could then be fed to a solver for determining the optimal GM weights to fit the outputs according least squares. However, our experiments have shown that this approach quickly becomes intractable, as requires too many samples to achieve fitting that both stable wrt. their selection and generalizes well locations that lie between selected points (see Appendix B for details).
To avoid the implied restrictive memory overhead and severe bottleneck would cause, chose to pursue a less resource-heavy solution and propose the following heuristic instead: First, coarse approximation (a') of the new weights is computed according
a if a, > 0
of
(6)
otherwise
where a; are the weights of the old mixture, and i runs through all Gaussians, The new weights a" then computed using correction based the transfer function
{gm{ B,, a, B, C)) ay' dl x gm( Bn a', B, C)
4

---

Published as a conference paper at ICLR 2022
Input
Target
Output
Input
Target
Output
(a) Overly smooth zero-crossing
(b) Bad covariance configuration
Figure 3: Failure cases for dense fitting: We show inputs, ReLU reference outputs, and our own. In (a), the zero crossing overly smooth, and the output weight too small. The positive Gaussian should have been moved up instead. (b), covariances have a configuration, they should have been reduced, and the position moved.
where B contains the positions, C the covariance matrices, and +p represents the ReLU, Effectively, the fraction computes correction term for the coarse approximation a'. is guaranteed, that the new weights a" are positive, just like we would expect from ReLU. Examples of the coarse approximation (a') are given Figure 2c, and the final fitting (a") in Figure 2d. The fitting most precise at the Gaussian centers. Problems arise at the discontinuity of the ReL.U, where the result can be overly smooth, as seen in Figure 3a, or with an adverse configuration covariances, as is the case in Figure 3b. Appendix C explains some of the design choices in more detail.
Reduction step. The next step in our fitting performs sxpectation-maximzaaiii number of Gaussians in the mixture. We implemented two variants, simpler modified EM variant based on the derivations from Vasconcelos Lippman (1999), and our own tree-based hierarchical EM method. For the simpler, modified EM variant, we select the N, Gaussians with the largest integrals out of the N, available ones. Then, one step of EM is performed, balancing accuracy with compute and memory resources, resulting in the fitting. However, this procedure implies O x Np) in terms of memory and computation complexity, rendering it too expensive for larger mixtures.
(EM) to reduce the
To remedy this issue, we propose tree-based hierarchical expectation maximization (TreeHEM). fundamental idea of Hre is to merge Gaussians in close proximity preferentially. To this end, Gaussians are first sorted into a tree hierarchy, based on the Morton code of their position, similar to Lauterbach et al. (2009). This tree is traversed bottom-up, collecting up to 27' Gaussians and performing a modified EM step, which reduces the number to T Gaussians. The fitted Gaussians are stored in tree nodes. The process continues iteratively, collecting again up to 27' Gaussians followed by fitting until we reach the root. Finally, the tree is traversed top-down, greedily collecting the Gaussians with most mass from the nodes until the desired number of fitted Gaussians is reached. Hence, TreeHEM yields an ON,, log N.) algorithm. Experiments given in Section 5.3 show, that value of T works best in terms of computation time, memory consumption, and often fitting eтror. Examples of the reduction are shown in Figure 2e. For more, details please see Appendix D.
4.2 REGULARIZATION AND MAINTAINING VALID KERNELS
order to add support for regularization to the learning process, we can apply weight decay, which pushes weights towards zero and the covariance matrices towards identity:
d, = a² + mcan (CC, - I) (C; I)),
(8)
where de the weight decay loss for kernel Gaussian and a, C, and are the weights, covariances, and the identity matrix, respectively, and @ is the Hadamard product.
Training the GMCN includes updates to the covariance matrices of kernel Gaussians. In order to prevent singular matrices, C" is learned, and then turned into symmetrical non-singular matrix by
= c C' Ic.
4.3 POOLING
In conventional CNNs, pooling merges data elements and reduces the (discrete) domain size at the same time. As filter kernels remain identical in size but are applied to a smaller domain, their receptive field effectively increases.
5

---

Published as a conference paper at ICLR 2022
conventional log softmax
GCL (CL(F,E,
128 Caussians
GCL, (1x8/64)
GCL (8x16/22
GCL (16x32/16)
GCL (32x64/0)
GCL (64x10/al))
integrate Gaussians
conventional BN
covariance norm
convolution F F.
ReLU N, components
Figure 4: The GMCN used in for our evaluation uses 5 Gaussian convolution layers (GCL, see right-hand side), integration of feature channel mixtures, and conventional transfer functions. See Section 5 for more details.
For pooling data with GMCNs, we can set up the fitting step (Section 4.1) such that the number of fitted Gaussians is reduced by half each convolution layer. While this effectively pools the data Gaussians by increasing their extent, it does not scale the domain. Thus, using the same kernels on the so-fitted data does not automatically lead an increased receptive field. To achieve this, the kernel Gaussians would have to increase in size to match the increase in the data Gaussians' extent. Instead, similar to conventional CNNs, we simply reduce the data mixtures' domain by scaling positions and covariances. We compute the scaling factor such that after each layer, the average trace of the covariance matrices becomes the number of dimensions (2 or currently), leading to an average variance of 1 per dimension. Effectively, this leads to Gaussian centers moving closer together and covariances becoming smaller. Hence, the receptive field of each kernel becomes larger relation to the original data scale, even if the kernel size is fixed.
5 EVALUATION
To evaluate our architecture, we use the proposed GMCN architecture to train classification networks on series of well-known tasks. We used an NVIDIA GeForce 2080Ti for training. For all our experiments, we used the same network architecture and training parameters shown in Figure 4. First, we fit a GM to each input 2D or 3D sample by using the k-means algorithm for defining the center of the Gaussians and one step of the EM algorithm compute the covariance matrices. The mixture weights are then normalized such that the mixtures have the same activation across all training samples on average. Covariance matrices and positions are normalized using the same procedure as for pooling (Section 4.3). The input GM is then processed by four consecutive blocks of Gaussian convolution and ReLU fitting layers. The numbers of feature channels are [8, 16, 32, 64] and the number of data Gaussians is halved in every layer. Finally, additional convolution and ReLU block outputs one feature channel per class. For performance reasons, we do not reduce the number of Gaussians in this last block. These feature channels then integrated to generate scalar value, which is further processed by a conventional batch- natch-normalization layer. The resulting values are converted to probabilities by using the Softmax operation, from which the negative-log- likelihood loss is computed.
The kernels of each convolutional layer are represented by five different Gaussians, with weights randomly initialized from a normal distribution with variance 1.0 and mean 0.1. One Gaussian is centered the origin, while the centroids of the remaining ones are initialized along the positive and negative x- and y-axes at distance of 2.5 units from the origin. For the 3D case, the remaining Gaussians are placed randomly at a distance of 2.5 units from the origin. The covariance factor matrix of each Gaussian is initialized following (I + R * 0.05) * 0.7, where I is the identity, and R a matrix of random values the range between -1 and 1. The covariance matrix was then computed according Equation 9. We determined these values using grid search. Examples of kernels are shown in Appendix E.
All models are trained using the Adam optimizer, with initial learning rate of 0.001. The learning rate is reduced by a scheduler once the accuracy plateaus. Moreover, we apply weight decay scaled by 0.1 of the learning rate to avoid overfitting, as outlined in Section 4.2.

---

Published as a conference paper at ICLR 2022
7
Figure 5: GM fitting examples of MNIST digits. 64 Gaussians were used represent each input sample.
Table I: ModelNet10 classification results of our method in comparison with PointNet and PointNet+++. We show two variants of competitor training: 1., on a set of points that matches the memory footprint of the GM, and 2., trained on the same GM inputs. The number of Gaussian (G) and the number of points (P) are indicated the header for each column.
/ P 64G/2133
128G / 426
#Params
PointNet
91.3 %
92.0 %
92.2 %
3.4 M
Points
PointNet++
92.8 %
93.3 %
93.9 %
1.4 M
PointNet
90.7 %
91.9 %
91.6% %
3.4 M
GM
PointNet++
91.8 %
92.1 %
92.4
1.4
Ours
92.1 %
92.3 %
93.3
215
5.1
2D DATA CLASSIFICATION (MNIST)
The MNIST data set (Lecun et al., 1998) consists 70 handwritten digits from 10 different classes. In this task, models have predict the digit each image represents, where performance is measured as overall accuracy. We trained our GMCN architecture using the standard train/test splits by fitting 64 Gaussians each image and processing them with our model. Figure shows some examples of the generated GMs. Our trained model achieved an accuracy of 99.4 % on this task, which is close to the 99.8 % currently reported by other srate-of-the-ee methods (Byerly et al., 2020; Mazzia et al., 2021). However, when compared to multi-layer perceptron (Simard et al., 2003) (98.4 %) or vanilla CNN architectures (Lecun et al., 1998) (99.2 %), which more closely reflect the GMCN design we used, our method performs better. Compared to point-based methods, our network achieves competitive performance (PointNet (Qi et al., 2017a) 99.2 % and PointNet++ (Qi et al., 2017b) 99.5 %).
5.2 3D DATA CLASSIFICATION (MODELNET 10)
ModelNet10 (Zhirong Wu et al., 2015) a data set used to measure 3D classification accuracy and consists of 3D models of different man-made objects. The data set contains 4.8 K different models from 10 different categories. We trained our network on ModelNet10 using the standard train/test splits without data augmentation. We used the point clouds provided by Qi et al. (2017b), which were obtained by randomly sampling the surface of the 3D models. We then fit 128 Gaussians to these point clouds generate the input for our network. Our trained model achieved an accuracy of 93.3 %, which is similar to other state-of-the-art classifiers: 94.0 % (Klokov & Lempitsky, 2017) or 95.7 % (Li et al., 2018a). Competing CNN-based methods reach 92% (Maturana & Scherer (2015)) and 91.5% (Riegler et al. (2017)). Appendix F provides additional, more detailed results.
In Table 1, we again compare our method with PointNet (Qi al., 2017a) and PointNet++ (Qi et al., 2017b), as they represent two well-established methods for point cloud processing. To enable fair comparison with each point-based technique, we ran two variants: The first uses a point cloud as input, with number of points equivalent to the memory footprint of our GM representation. The second uses the same GM inputs as our network, where weight and covariance matrix are processed as additional features for each point. Models are trained with various input sizes: 32 Gaussians (106 points), 64 Gaussians (213 points), and 128 Gaussians (426 points). We found that our method outperforms PointNet all experiments using order of magnitude fewer parameters. It also achieves a similar performance to the point-based hierarchical architecture PointNet++ using five times fewer parameters. Our method outperforms both architectures when processing GMs directly.
7

---

Published as a conference paper at ICLR 2022
k-means init
k-means + EM
random + EM
Eckart
Figure GM fitting examples ModelNet object with 128 Gaussians for k-means and EM algorithms and 125 Gaussians for Eckart et al. (2016b). k-means initialization method does not produce smooth surface fittings, but Gaussians have uniform covariances. EM with k-means produces results that are relatively sharp on edges and smooth surfaces. EM with random initialization and Fckart produces even smoother surfaces.
Table 2: Accuracy and training time on the ModelNet10 data set for our model wrt. the number of layers. Our network benefits from additional layers. However, accuracy saturates after five layers. While training times in our proof-of concept system are not competitive, GMCNs leave much room for optimizations future work.
2
3
5
6
Accuracy Training time
72.5 %
89.4 %
91.2 %
92.0
92.6 %
92.3 %
0.5h
2h
5h
10h
21h
41h
5.3 ABLATION STUDIES
In this section, we discuss additional experiments carried out to validate the individual elements of our network. For these ablation studies, we use the ModelNet10 data set all experiments.
Network depth. First, we evaluate the network's performance wrt. the depth of our GMCN to determine the optimal number layers. Table 2 shows the classification accuracy on the ModelNet10 data set for different models with varying numbers of convolution layers. We confirmed that GMCN can benefit from additional layers; however, the accuracy stops increasing once we reach five layers.
Gaussian mixture fitting. We evaluated the two proposed fitting algorithms (Section 4.1) used after each convolution layer. Three different reduction variants were tested: modified EM, TreeHEM with 7 and 4 Gaussians per node. Table 3 shows the results of these methods terms of the fitting error in the first three layers and the accuracy of the model when using 64 and 128 Gaussians as input, respectively. TreeHEM obtains lower fitting error in most of the layers when compared to the EM algorithm. Moreover, the tree-based method also requires less memory during training compared to EM, which runs out of memory when trained on inputs represented by 128 Gaussians. However, EM obtains slightly higher classification accuracy for 64 input Gaussian. Out of the two TreeHEM variants, the T performs better in terms compute time, memory, fitting error, and classification accuracy. Memory and time easily explained by the number Gaussians cached in the tree. We infer, that accuracy is reduced in higher T settings due to smearing of Gaussians (i.e., loss of fine details), which is more likely to occur when more of them are merged simultaneously.
Input data fitting. Lastly, we evaluate how the model is affected by the different algorithms used fit the input data. We compare four different methods: k-means initialization plus one single step of
Table 3: Evaluation of our proposed fitting algorithm. We find that TreeHEM achieves lower errors than the modified EM algorithm when fitting the output of different layers. Nonetheless, accuracy is similar. Moreover. we sec that the EM algorithm runs out of memory when using 128 input Gaussians and batch size of 21.
RMSE per Layer (#G=64)
# Gaus.
Memory
Training time
2
3
64
128
64
128
64
128
EM
0.45
0.50
0.36
92.2 %
n/a
2.5GB
n/a
6h
n/a
Tree (4)
0.35
0.22
0.25
91.3 %
92.9 %
1.6GB
6.8GB
5h
22h
Tree (2)
0.36
0.21
0.22
92.0 %
93.4 %
1.3GB
5.6GB
4h
19h
8

---

Published as a conference paper at ICLR 2022
Table 4: Accuracy of our model on the ModelNeet0 data for input GM generated using different algorithms. Results show that k-means produces the best accuracy, while more advanced methods that generate uneven covariance matrices, such as Eckart et al. (2016b), reduce the classification accuracy of the network.
k-means
k-means+ EM
rand+EM
Eckart
93.3 %
92.1
92.8 %
91.0% %
EM (k-means), k-means initialization plus multiple EM steps (k-means+EM) random initialization plus multiple EM steps ErM) and the method proposed by Eckart et al. (2016b) (Eckart). Figure 6 shows the resulting GMs of the four different algorithms for two different models from ModelNet10. We see that k-means produces set of Gaussians with relatively uniform shapes, but overall does not fit the data accurately. Random init + EM and Eckart can fit the model with high precision but also produces some blurred areas. K-means init EM achieves the best representation of the four methods.
The test accuracy of our network for the four types of input GMs is reported in Table 4. K-means initialization achieves the best accuracy while Eckart performs worst. We believe these results can be explained by the uneven Gaussian covariance matrices in methods such as Eckart and rand+EM.
6 DISCUSSION AND CONCLUSION
To the best of our knowledge, we are the first to propose deep learning method based on the analytical convolution of GMs with unconstrained positions and covariance matrices. We have outlined complete architecture for the task of general-purpose deep learning, including convolution layers, a transfer function, and pooling. Our evaluation demonstrates that this architecture is capable of obtaining competitive results well-known data sets. GMCNs provide the necessary prerequisites for avoiding the curse of dimensionality (Appendix A). Hence, we consider our architecture as a promising candidate for applications that require higher-dimensional input compact networks.
Limitation and future work. Currently, the most significant bottleneck limit the universal application of our work is the the--uavvidable-fitti
process. As reflected by the results in Table 2, the implied computation time limits the size of GMCNs that can be trained with consumer- grade hardware in appropriate amount of time. For the same reason, data augmentation was omitted in our tests. The basic architecture presented in Section 5 is insufficient for achieving competitive results more complex tasks, such classification on ModelNet40 (details in Appendix F.2) and MashionMNIST (86% accuracy, others more than 90% (Xiao et al., 2017; zalandoresearch, 2019)).
However, early experiments have shown that performance can be drastically improved by embedding a fitting step in the convolution stage. Specifically, we can use an on-the-fly convolution of the positions construct auxiliary tree data structure, in manner similar to Section 4.1. The resulting tree is then used to perform full convolution and fitting in one stage. We have identified highly parallel algorithms for the required steps and confirmed that applying these optimizations to our methods can significantly reduce memory traffic and improve computation times by more than an order of magnitude.
We further conjecture that the achieved prediction quality with our architecture is currently limited by fitting accuracy (see Tables 3, 10, 11 and Appendix F). This could be addressed by sampling the GM and performing one two EM steps on these samples. Doing so should become feasible once the number of Gaussians is brought down using the convolution fitting step described above. Presumably, slight increase in accuracy would enable us to achieve state-of-the-ar results all tested use cases.
Additional extensions to raise the applicability of GMCNs would be the introduction of transposed convolution, as well as skip connections (Romneberger et al., 2015). We expect that these tasks could be solved with reasonable amount of additional effort. We hope to enable novel solutions for 3-dimensional problems with these extended GMCNs in future work, such as hole- dilliinnng for incomplete 3D scans learning physical simulations for smoke and fluids.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY
The source code for a proof-of-concept implementation, instructions, and benchmark
datasets are provided in GitHub repository (https://githan com/ wwt.ccccccoooo.eeeeccceeeoooeeoe tugteeiite/ Gaussian Coussin-MMxtuue--nnn Mixture-Convolutin -ettonnnet)t)))
ACKNOWLEDGMENTS
The authors would like to thank David Madl, Philipp Erler, David Hahn and all other colleagues for insightful discussions, Simon Fraiss for implementing fitting methods for 3d point clouds. We also acknowledge the computational resources provided by TU Wien. This work was in part funded by European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska- Curie grant agreement No 813170, the Research Cluster "Smart Communities and Technologies (Smart CT)" TU Wien, and the Deutsche Forschungsgemeinscaatt (DFG) under grant 391088465 (ProLint).
REFERENCES
Eman Ahmed, Alexandre Saint, Abd El Rahman Shabayek, Kseniya Cherenkova, Rig Das, Gleb Gusev, Djamila Aouada, and Bjorn Ottersten. A survey on Deep Learning Advances on Different 3D Data Representations. 1(1):1-35, 2018. URL http: /arxiv org/abs http//ass11000 1808, 01462.
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by extension operators. ACM Transactions on Graphics, 37(4), 2018. ISSN 15577368. doi: 10.1 10.1145/3197517. 3201301.
Alexandre Boulch. ConvPoint: continuous convolutions for point cloud processing. pp. 1-12, 2019.
URL aattp://arkee org/ abs/aa////1../ 02375.
David S Broomhead and David Lowe. Radial basis functions, multi-variable functional interpolation and adaptive networks. Technical report, Royal Signals and Radar Establishment Malvern (United Kingdom), 1988.
Adam Byerly, Tatiana Kalganova, and Ian Dear. A branching and merging convolutional network with homogeneous filter capsules, 2020.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1977. doi: 10.1111/j.5117-6666 197.1.977..6160000
Ben Eckart, Kihwan Kim, Alejandro Troccoli, Alonzo Kelly, and Jan Kautz. Accelerated generative models for 3D point cloud data. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem:5597-5555, 2016a. ISSN 10636919. doi: 10.1 10.1109/CVPR.2016.500
Benjamin Fckart, Kihwan Kim, Alejandro Troccoli, Alonzo Kelly, and Jan Kautz. Accelerated generative models for 3D point cloud data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5497-5505, 2016b.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep Learning for 3D Point Clouds A Survey. IEEE Transactions on Pattern Analysis and Machine In- telligence, pp. 1-24, 2019. URL https:/ /ww.///oiiiioo org/ 1t..g/0..1110 1ww.111111PPA
2020 3005433..
Pedro Hermosilla, Tobias Ritschel, Pere Pau Vázquez, Alvar Vinacua, and Timo Ropinski. Monte Carlo convolution for learning on non-uniformly sampled point clouds. SIGGRAPH Asia 2018 Technical Papers. SIGGRAPH Asia 2018, 37(6), 2018. doi: 10.1145/377212::225511..
Maximization Wenzel Jakob, Christian Regg, and Wojciech Jarosz. Progressive Expectation Expecaaion-Maxiiizatioon for Hierarchical Volumetric Photon Mapping. Computer Graphics Forum, 30(4):1287-1229) jun 2011. ISSN 01677055. doi: 10. 1 1467-8659.7788 89177-8659.201101988888. URL tttp:: //////dd /ooi. wiley. com/
10. 14 67 8699 201 01988.
10

---

Published as a conference paper at ICLR 2022
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3D Convolutional Neural Networks for Human Action Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):221-231, jan 2013. ISSN 0162-8828. doi: 10. 10.1109/PPAM.2202..55 URL AttLttt/eeeeeepp.oo ieee. org/document/ / 61 65309/.
Roman Klokov and Victor Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolu- tional neural networks. Advances Neural Information Processing Systems 25, 25:1097-1100 may 2012. ISSN 0001-0782. doi: 10.1 10.1145/3065386 URL https: /ttp///..//// acm. org doi/10. 1145/3065386.
C. Lauterbach, M. Garland, S. Sengupta, D. Luebke, and D. Manocha. Fast BVH construction on GPUs. Computer Graphics Forum, 28(2):75--34,, 2009, ISSN 01677055. doi: 10.111
77778865..2009001777
Allan M. M. Leal. autodiff, modern, fast and expressive C++ library for automatic differentiation, 2018. URL https: autooiff..ffffooooooo www.ooifff..ggg.hbb.ooo.. github..h...... io.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-2222 1998. ISSN 001 89219. doi: 10.1 10.15 726791. URL http: htt//eeeeepppooree Leee org.org//ooooooooooo http//docmnnttt//2 772679///
Jiaxin Li, Ben M. Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In 2018 TEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018 8a.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. PointCNN: Convolution On x- T-aansformed Points. Advances in Neural Information Processing Systems, 2018-Decem: 820-830, jan 2018b. ISSN 10495258. URL http: : ataaaaiiiii///../ org/
077911
Daniel Maturana and Sebastian Scherer. VoxNet: A 3D Convolutional Neural Network for real- time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 222-928. IEEE, sep 2015. ISBN 978-1-4799-999941 7353481. URL http: : httptttttteeeeeeettooooc httpeeeeeeppporreeeeeeeeeeeeeeee www.eepwpoe............e leee org/ http://ommeeettte333eeeee
doi: 10.1 10.1109//ROS.200 2015.
Vittorio Mazzia, Francesco Salvetti, and Marcello Chiaberge. Efficient-capsnet Capsule network with self-attention routing. arXiv preprint 12441
Reinhold Preiner, Oliver Mattausch, Murat Arikan, Renato Pajarola, and Michael Wimmer. Contin- uous projection for fast reconstruction. ACM Transactions on Graphics, 33(4):1-13, 2014.
ISSN 07300301. doi: 10. 1 000114//20019972660
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet:
Deep Learning on Point Sets for 3D Classification and Segmentation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017a. doi: 10.1109/CVP..2017..6
Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets Metric Space. (Nips), jun 2017b. doi: 10. 1001:///0039-1114990011111011110 URL http:://azz
oro.orggo//..o 706..0666 02413.
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. OctNet: Learning deep 3D representations at high resolutions. Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, volume 2017-Janua, 2017. ISBN 9781538604571. doi: 10.1109/CVPR. 2017.701.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted In- tervention - MICCAI 2015, pp. 234-241, 2015. doi: 0000077978-3331-24455 28. URL http:// http://lint 1iik..n.... aepringer... com 10. 100. 000000099783331111 3--31---111--------
P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings., 2003.

---

Published as a conference paper at ICLR 2022
Hugues Thomas, Charles R. Qi, Jlann-Emmanue Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. KPConv: Flexible and Deformable Convolution for Point Clouds. apr 2019. URL http: /atp://aaaxxi www. C8889.
Nuno Vasconcelos and Andrew Lippman. Learning mixture hierarchies. Advances in Neural Information Processing Systems, pp. 606-612, 1999. ISSN 10495258.
Susana Vinga. Convolution integrals of Normal distribution functions. Supplemen- tary material to S. Vinga and JS.Almeida (2004) "Renyi continuous entropy of DNA se- quences". J.Theor.Boll 237(3):377-388, 2004. URL http:://wb.ist.ut...2.
Peng Shuai Wang. Yang Liu, Yu Xiao Guo, Chun Yu Sun, and Xin Tong. O-CNN: Dctree-based convolutional neural networks for 3D shape analysis. ACM Transactions Graphics, volume 36, 2017. doi: 10. I 144/3072959.3073666
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint (XXv:1708.007477 2017.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters. In Computer Vision ECCV 2018, pp. 90-105, mar 2018. doi: 10. 10.000779788--000-111333333 6. URL 10 10010000//90 /0000//00000000--00--01---11--11
org http://ggggbb////oeeee 11527htttt
zalandoresearch. Fashion mnist benchmarks, 2019. URL https: wtt....ccomm
salandoresearcc/ffaahioo com-mmnissttt #benchmark.
Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: deep representation for volumetric shapes. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912-1920. IEEE, jun 2015. ISBN 978-1-4673-6964-0. doi: 10. 00.110991P..2005522888.... URL http: wwweeeeeeeeooeeeeeeeeeee ieee. htt:/ddcuumnt/72980001
12

---

Published as a conference paper at ICLR 2022
A THEORETICAL MINIMAL MEMORY FOOTPRINT (AND PARAMETER COUNT)
This section provides formal equations and concrete examples to outline the theoretical minimum memory footprint of GMCN and enable assessment of its overall potential for reducing memory consumption. For this, we made the following assumptions:
None of the modules (convolution/rreduction fitting, transfer function fitting, pooling) rely on caches.
Programming code not counted.
Minimal storage requires 6 floats for 2D Gaussian, and 10 floats for 3D Gaussian.
Fitting can be integrated into the convolution layer (see the following paragraph) and produces a mixture with specified number of Gaussians N.. No fitting performed if the specified number is equal to the number of Gaussians produced by the convolution.
In conventional CNNs, the convolution operation, that is "per-pixel kernel-sized window, element- wise multiplication, and sum", is usually built into one compact routine, CUDA kernel. Ac- cordingly, intermediate results are never stored in memory. The proof-of-concepl implementation described in this paper is not compact in the same way. The closed-form solution of the convolution with GMCNs yields a large number intermediate output Gaussians, which are then greatly reduced in the fitting step before being forwarded to the next layer. The described prototype explicitly stores these intermediate results, making it easier to implement and explain. As outlined in Chapter 6, the integration of fitting step directly into the convolution is possible with our approach, albeit more complex compared to conventional CNNs. However, in comparison to conventional CNNs, such compact implementation of GMCNs can achieve significant savings on memory footprint.
Integrated convolution-fitte fittiingi module A convolution module has F; input feature channels and F, output feature channels, with batch size set to B. The memory consumed by the incoming data is accounted towards the previous module. Given incoming data mixtures containing N; Gaussians and kernels containing Ng Gaussians, our proof-of-concept implementation produces and stores N, N₁ Gaussians, which will be reduced before pooling is applied. In contrast, an integrated convolution fitting module can produce a smaller number of Gaussians No directly. Ideally, No 2Np, such that the the subsequent pooling layer can select half of the provided inputs to yield Np output Gaussians.
The minimal memory footprint in terms of Gaussians (G) is the sum of kernels (K), and data (D) times 2 (for the gradient):
K = F,FON
D B(F.N, F.N,)
G = 2(K + D).
(10)
K can also be used to compute the number of parameters of the given module by multiplying with the representation used (6/10 floats for 2D/3D, respectively).
Minimal memory footprint for trained model The factors and results for the model used for our evaluation (shown Figure 4) are given in Tables and 6. Overall, N, N, used, the
Table S: Minimal memory footprint for the network shown Figure 1, using Equations 10, if N. = 2N,. The total number of Gaussians for each convolution module is given as C. Theoretical memory usage in megabytes is given in columns Mip and MsD, using 2D and 3D Gaussians, respectively.
B
F
F
N;
N₂
N,
N₂
K
D
G
M2p
M3D
32
1
128
128
64
5
40
49,152
98,384
2.25
3.75
32
8
16
64
64
32
5
640
49,152
99,584
2.28
3.80
32
16
32
32
32
16
5
2,560
49,152
103,424
2.37
3.95
32
32
64
16
16
8
5
10,240
49,152
118,784
2.72
4.53
32
64
10
8
4
5
3,200
3.840
14,080
0.32
0.54
13

---

Published as a conference paper at ICLR 2022
Table 6: Minimal memory footprint for network shown in Figure 1. using Equations 10, if implemented without combined convolution-fitiee fitting. The total number of Gaussians for each convolution module given G. Theoretical memory usage in megabytes is given in columns M2D and MaD.
B
Fo
N
No
N₂
N
D
G
M2D
MaD
32
128
640
64
40
180,224
360,528
8.25
13.75
32
8
16
64
2,560
32
5
640
1,327,104
2,655,488
60.78
101.30
32
16
32
32
2,560
16
2,560
2,637,824
5,280,768
120.87
201.45
32
32
64
16
2,560
5
10,240
5,259,264
10,539,008
241.22
402.03
32
64
10
8
2,560
4
5
3,200
820,480
1,647,360
37.71
62.84
model has a theoretical minimal memory footprint of 434,256 total 3D Gaussians, which is less than 17 megabytes. In comparison, the proof-of-concept implementation presented in this paper has a theoretical use of 781 megabytes. In practice, it still requires several times more, since it uses exhaustively optimized.
convenience data structures that
B COMPARISON OF LEAST-SQUARES SOLUTION AND HEURISTIC FOR RELU FITTING
In this section, we will compare least-squares solution with our heuristic presented in Section 4.1 for the fitting of transfer function.
The least-squares solution has to solve the following problem
Ay = t,
(11)
where t, (p(gm(D)) are the target values at positions and is matrix:
(Do)
(N()
A
(12)
20 (DN) NN (NN)
with (xx))= V(2n)* det (C)
(a ".)",," Cr-'0x-8 (x-B.), B, is the centroid, C, the covariance of Gaus-
sian is the number of Gaussians, and the number of dimensions (e.g., 3). other words, A contains the evaluations of the mixture Gaussians with the weight set to one, on set of points
contained in D.
We evaluated several choices for D):
= B, evaluate the mixture at the centre points of its Gaussians
rand(M [M, k), positions within the domain
concat( B, rand(M, k)), mixture centres and random positions within the domain
Figure shows part of test input, along with the result of our heuristic. After seeing the results in Figure 8 and timings in Table 7, we refrained from computing the RMSE and selected the heuristic for network. The least-squares solution overshoots away from the sampling points, and the compute resources are several magnitudes higher compared to our heuristic.
The matrix A with choice B can be of rank(A) N, and therefore (non-pseudo) inversion methods are not applicable.
14

---

Published as a conference paper at ICLR 2022
Table 7: Computation time for various fitting methods data set containing 800 mixtures, 256 Gaussians each. torch. win.lg..inn piny and torch. wiiaalgg........................ 1st sq were tested, both functions being able solve our problem. However, isteq does not compute a gradient in the current pytorch version and would be therefore unusable. But even if so, the timings are prohibitive. Our heuristic takes only 3.8 milliseconds.
D=
B
concat (B, rand N, k))
rond k)
torch. linalg pinv torch. linalg. 1stsq
10.2s
10.9s
11.3s
1.1s
1.5s
2.0s
Input
Target (input GM)
Heuristic
Figure 7: Selection of the 800 Gaussian mixtures used for the evaluation of the ReL.U fitting method.
15

---

Published as a conference paper at ICLR 2022
D= rond(N; (N, k))
D rand(2N, k)
D= B
concot B, rond(N,R) k))
squares algorithm. verified that mixture fits the target at sample positions, so the solution the least-squares problem correct in all cases. The mixture overshoots however and does not fit the ReLU well away from the sampling positions. Increasing the sample count, or selecting hetter samples helps, but even then there is no guarantee that the mixture won't overshoot. torch. 1 inalg. 1stsg was used, but the results for torch. Linalg. pinv similar.
Figure GM fitting using
16

---

Published as a conference paper at ICLR 2022
C DETAILS ON THE DENSE FITTING
The purpose of the dense fitting step in our approach is to fit a transfer function, for example, a ReLU. Our fitting method only computes new weights for the Gaussian mixture. Position and covariance matrices of the Gaussians are not changed (as described in Section 4.1). Accordingly, we give only equations for the new weights a", which are computed in steps.
Coarse fitting. The first step computes coarse, or initial approximation to the target (f(gm)):
0is if a; > 0 E, otherwise
df
(13)
where ag are the weights of the old mixture, and i runs through all Gaussians. This design is motivated by two key considcrations.
First, we argue that it is safer if the weights in the fitting are strictly positive. In our design, this will be the case, since a' > and the second step does not introduce a negative sign. Negative weights themselves are not problem for the GMCN architecture. However, the ReLU defines function that is positive throughout the domain, and so should the fitting. This in itself still does not conflict with negative weights as long the sum is greater than zero for all positions in the domain. However, approach only considers the mixture's ' evaluation at the Gaussians' centroids, and negative weights could cause the sum to become negative locations in-between these evaluated points. Something similar happens when using a linear solver (Section B). As positive side effect, reduction fitting becomes simpler if does not have to deal with negative weights.
Secondly, we scale the Gaussians (change their weight) as little as possible, therefore we do not initialize at simply to 1. The rationale here is, that we want to retain the shape of the mixture as much as possible without having to fetch the positions and covariance matrices from memory. Now, we initialized the weights to I, the result at the centroid would be approximately correct, but the shape would change more than necessary.
Correction
The new weights a" are computed using a correction based on the transfer function (gm( Bi a, B)C)
a
(14)
gm B, B,C)
where B contains the positions, C' the covariance matrices, and represents the ReLU, The fraction computes a correction factor for each Gaussian by evaluating the target function and the current approximation at the Gaussians centroid. This computation is O(N²) in the number of Gaussian components in terms of runtime. In order to avoid storage of intermediate results and large gradient tensor in PyTorch, we implemented the evaluation in a custom CUDA kernel.
On the CPU it would be possible trade accuracy for performance by ignoring contributions that are below a certain threshold, and using a tree-based acceleration structure (e.g., bounding volume hierarchy). However, our experiments have shown, that this does not scale well on the GPU. The evaluation kernel is reasonably optimized. Still, due to a large number of Gaussians, which can easily go into several thousands, times the number of feature channels, times the batch size, these two evaluations account for roughly half of the overall training/test runtime.
In order to improve performance decisively, either the number of Gaussians would have to be reduced before this fitting step (c.g., during the convolution as outlined in our future work section), different algorithm could be used, or both.
D DETAILS ON THE REDUCTION STEP
The purpose of this step in our approach is to reduce the amount of data by fitting the target with a smaller mixture. It is applied after ReLU fitting, hence all weights are positive, and the mixture can be turned into a probability distribution by dividing the weights by their sum. Therefore, probability- based methods can be applied. In particular, it would be possible to sample the mixture and then use an expectation-maxiine the potentially large number of Gaussians.
(EM) algorithm. However, that would be prohibitively expensive, given
17

---

Published as a conference paper at ICLR 2022
Since TrecHEM uses equations from the hierarchical EM algorithm by Vasconcelos & Lippman (1999), we start our detailed description with HEM and then continue with details on TreeHEM.
Hierarchical EM Hierarchical EM starts with one Gaussian per point, which is the first level. In each following level (I + 1), a smaller mixture is fitted to the previous level (l). The new mixture needs to be initialized first, for instance by taking random subset of the mixture in level l. In the step, responsibilities are computed using
[(O, 10." (9 )c,
Fia Ex [(0 [(0 10 )
1(0!+)100 (o -
(15)
where úi; is the number of virtual samples (equations taken from Preiner et al. (2014)). In the original work, N points were fitted, so dē;- = Nu corresponded to the number of points that Gaussian was representing. Since we do not process points but Gaussians directly, N becomes an implementation defined constant, Afterward, the M step can be taken:
w (1+1)
Σ riaw!
wis Tio ////////t ((+1)
a++) = Σ wisli
0(++1
Σ Wis (x + (u, u ((u - 1 (
(16)
where w, 1e, and Σ are the weight, centroid, and covariance matrix, respectively, i the index of the (larger) target mixture, the index of the new mixture. In each level, only one iteration is taken (E and M step), and the algorithm continues with the next level. Preiner et al. (2014) propose a geometrical regularization, in which ris is set to zero if the Kullback-Leibler divergence between Gaussians and 8 is larger than a certain threshold.
We were not able to use hierarchical EM due to multiple reasons, including difficult parallelization on the GPU, low test accuracy on our data, poor performance (since every Gaussian needs to be tested against each other Gaussian), and the memory footprint of the corresponding gradient.
We, therefore, developed the TreeHEM algorithm to improve on these issues. In short, build spatial tree of the (larger) target mixture and use it to localize the fitting, i.e., limit the fitting to a fixed-size subset of Gaussians in proximity of each other.
TreeHEM
The tree is built using the same algorithm, as is described by Lauterbach al. (2009) for building bounding volume hierarchies of triangles (LBVH), without the bounding boxes. In our case, we take the Gaussians' centroids, compute their Morton code, and use it sort the mixture. This puts the Gaussians on Z-order curve. It is then possible to build a tree in parallel by looking at the bit pattern of the sorted Morton codes. Please consider the excellent paper from Lauterbach et al. for more details. Storing the indices of children and parents in the tree allows us to traverse bottom-up and top-down.
The tree is traversed twice. In the first, bottom-up, pass we collect and fit Gaussians, and then store them in a cache with up T Gaussians in the tree nodes. Bottom-up traversal is implemented in parallel way. Threads in a thread pool start from leaf nodes. The first thread reaching an inner node simply sets flag and is re-entered into the pool. The second thread to reach an inner node does the actual work: First, the Gaussians from the caches of the two children are collected, resulting in up to 2T' Gaussians. If the children are leaves, or not yet full, then that number can be below T, in which case they written into the cache. Otherwise, T new Gaussians are fitted to the collected ones, stored in the cache, and traversal continues until reaching the root node.
For the fitting, we first select T initial Gaussians from the 27 collected ones. This is done by dividing the 2T Gaussians into T clusters based on the Euclidean distance between the Gaussians' centroids. From each cluster, the Gaussian with the largest weight is selected, giving T initial Gaussians. The
18

---

Published as a conference paper at ICLR 2022
Initialization
After about million training steps
Figure 9: Convolution kernels of a 2D data set. It is clearly visible, that all of the parameters (weights, positions, and covariance matrices) adapt to the data, once training progresses.
rationale behind this method is to to select the most important Gaussians that have the least overlap. We then use Equations 15 for the step, and Equation 16 for the M step the fitting, where we set I ++ 1 to the initial mixture described above, and to the target mixture consisting of 27 Gaussians.
At this point, we obtain tree with mixture In the second, top-down, pass we aim to select N/T nodes that best describe the target mixture, where is the number Gaussians in the fitting. This was implemented using a greedy algorithm with single thread per mixture (which is not problem the kernel only traverses the very top part of the tree). All selected nodes are stored In each iteration, of Gaussians, i.e., sum of weights) and replace it with its two children. This adds 7 nodes to our selection in each iteration. The iteration terminates once we selected N/T nodes. Finally, we iterate through these nodes and copy the Gaussians from the cache into the destination tensor.
Gaussians in each node that fits the data underneath.
vector, where the first selected node is the root. search through the vector and select the node with the largest mass (integral
Gradient computation Since all of the above is implemented in a custom CUDA kernel, we do not have the usual automatic gradient computation. Therefore we had compute the gradient manually. To this end, we implemented unit tests of our manual gradient, which compare our manual gradient against a gradient computed by the autodiff C++ library (Leal, 2018).
E KERNEL EXAMPLES
Figure shows examples of training kernels for 2D data set.
F FURTHER RESULTS AND ABLATION
In this section, we include the full suite of results collected on the ModelNet10 and ModelNet40 data sets. Unless stated otherwise, the following tables report the test accuracy as shown in TensorBoard with the smoothing the fitting of the input data. As part of the ablation, we consider different network configurations and lengths. The network notation indicates Gaussian convolution layers (convolution, activation, and pooling) using arrows (-) and specifies the data dimensions between these layers by the number of feature channels / number of data Gaussians.
to 0.9. Times shown are for training on an NVIDIA GeForce 2080Ti, without
19

---

Published as a conference paper at ICLR 2022
Table 8: ModelNet10 with TreeHem(TT fitting and batch size 14, evaluating varying network lengths.
Epochs / acc.
Model Layout
50
60
#kernels
#params
mem
tr. time (60)
1/128 10
74.73
72.45
10
651
0.3GB
Oh 36m
1/128 + 8/64 - 10
89.37
89.36
88
5721
0.3GB
2h 6m
1/128 + - 16/32 - 10
91.06
91.17
296
19241
0.7GB
4h 49m
1/128 + 332/16 - 10
91.83
91.99
968
62921
1.6GB
10h 10m
1/128 + -64/ + 10
92.44
92.61
3336
216841
3,4GB
21h 22m
1/128 ... 7 128/4 10
92.22
92.30
12168
790921
5.9GB
41h 22m
Table 9: ModelNet10 with TreeHem(T fitting batch size testing number input Gaussians, all layouts, the number of kernels is 3,336, and the number parameters is 216.841.
Model Layout
Accuracy
1/128 + 8/64 + 16/32 32/16 64/8 + 10
92.78
1/64 8/64 16/32 - 32/16
64/8 10
92.28
1/32 + 8/64 + 16/32 + 32/16 + 64/8 - 10
91.44
1/32 8/32 16/32 #3/116
64/8 10
92.09
1/16 + 8/64 + 16/32 + 32/16 + 64/8 - 10
90.92
1/16 + 8/16 16/16 32/16
64/8 > 10
89.91
1/16 + 8/8 + 16/4 - 32/2 + 64/2 - 10
90.17
F.1 MODELNET10
In Table 8, we report the classification accuracy of different GMCN models that was achieved after 50/60 epochs, respectively, well their resource consumption. The simplest tested design uses a single feature channel and 128 fitted Gaussians for the input layer. followed by a 10-way classification. With the addition of convolution layers, the classification performance improves. The first internal convolution layer uses 8 input feature channels and 64 fitted Gaussians, which suffices to significantly raise accuracy. To test longer networks, each row in the table builds on the last and inserts an additional convolution layer just before the final classification layer. With each new layer, the number feature channels is doubled and the number of fitted Gaussians halved. The results demonstrate the GMCNs of variable length exhibit consistent behavior. After 50 training epochs, the classification accuracy no longer changes drastically and usually increases slightly given another 10 epochs. Classification accuracy eventually peaks using four internal convolutional layers and slightly drops when a fifth layer is added.
In Table 9, investigate the effect of varying the number of fitted Gaussians in each layer classification accuracy. Starting from the best-performing model identified previously with four internal convolution layers, we assess how adapting the internal fitting can influence classification accuracy. Naturally, the network is sensitive to changing the amount of information it receives, i.c., the number of Gaussians fitted the input data. However, the same is generally true for the internal layers. The GMCN architecture appears to favor distinct numerical relations between fitted Gaussians in the individual layers. For example, quartering the number of input Gaussians (32 vs. 128) without adapting the remainder of the network yields poorer performance than using only half as many fitted Gaussians in the first convolution layer (32 vs. 64).
In Table 10, we compare the different available versions for expectation maximization (EM) in the fitting of Gaussians: modified EM (MEM), and two variations of TreeHEM, with parameters = and T= 4, respectively. These results are obtained using data sets with 64 fitted input Gaussians. We examine the error the two fitting processes (dense fitting of the transfer function, and reduction / pooling) in three measurements. Mcasurement is done by computing the RMSE evaluations from several 100k sampled positions of the fitted GM and the input to the respective fitting process. Our measurements are:
20

---

Published as a conference paper at ICLR 2022
Table 10: Comparing different fitting methods 1/64 8/32 - 16/16 + 32/8 10 GMCN with batch size
21 ModelNetIO. We report the RMSE each fitting against ground truth in each layer.
RMSE
Layer
Method
all
ReLU
reduction
MEM
0,4457
0,1639
0,2653
Convolution Layer 1 + 8
TrecHEM (T=2)
0.359
0,2253
0.1144
TreeHEM (T=4)
0.3487
0.2204
0.1159
MEM
0,4991
0,1433
0,2694
Convolution Layer 8 + 16
TreeHEM (T=2)
0.208
0,1031
0,0717
TreeHEM (T=4)
0.223
0.1
0.0806
MEM
0.3582
0.1792
0,0842
Convolution Layer 16 - 32
TreeHEM (T=2)
0.2206
0.1346
0.0469
TreeHEM (T=4)
0.2461
0.1259
0.0707
all: overall fitting error, compares the output of the reduction step with ground truth transfer function evaluations of the input.
ReLU: compares the output of the dense fitting step with ground truth transfer function evaluations of the input.
reduction: compares the output of the reduction step with the input to the reduction step (ouput of the dense fitting).
Comparing the two, we find that the ReLU fitting error is significantly larger in each layer. This suggests that GMCN performance is affected more by the fitting accuracy than by the reduction of Gaussians. Note that in general, all + ReLU + reduction.
Table 11 confirms this trend when using input data sets with 128 fitted input Gaussians. However, it also illustrates our preference for using the TreeHem fitting algorithm using T = 2. The MEM method runs out of memory with this configuration, and therefore the errors cannot be evaluated. Comparing TreeHEM with T = and T 4, we find that the first variant, in addition to reduced resource requirements, yields both lower fitting error and better accuracy. As stated before, we attribute this behavior to the smearing of Gaussians when a larger parameter used (more Gaussians merged simultaneously and thus fine details can be lost).
Table 12 illustrates the development of the test accuracy with the number of epochs allowed for fitting Gaussians to the 3D input data set. These experiments mainly serve to identify the suitable number of fitting iterations to be used in our evaluation. A baseline 80% is quickly achieved with all approaches after only 5 epochs. Between 50 and 60 epochs, there is little discernible improvement of the accuracy. In terms of the fitting techniques to use, we find that all cases, k-means performed best for the ModelNet10 data set, further solidifying our previous results.
F.2 MODELNET40
In order assess GMCNs in the context of problem with higher complexity, we ran a series of tests for the ModelNer40 data set. Due to its larger size and the fact that our proof-of-concept GMCN implementation not heavily optimized, the number of experiments and amount of hyperparameter exploration that could be performed in a timely manner was limited. Data augmentation was not possible, because that would multiply the number of training samples and therefore runtime. Out of the tested variants, GMCNs performed best with model that contains four internal Gaussian convolution layers, similar to our findings for ModelNet10. The particular configuration we used is similar, with the exception that the last layer performs 40-way classification (1/128 + 8/64 16/32 - 32/16 > 64/8 - 40). Table 13 lists the achieved test accuracy with different batch sizes and epochs. This data confirms that GMCNs converged: going from 50 to 60 epochs only has a small effect on accuracy, improving it slightly. The best performance was achieved with a test accuracy of 87.6%, using batch size of 14, training for 60 epochs.
21

---

Published as a conference paper at ICLR 2022
Table 11: Comparing different fitting methods in a 1/128 - 8/4 + /6/32 - 32/16 - 64/8 - 10 GMCN with batch size 21 on ModeINet10. We report the RMSE of cach fitting against the ground truth in cach layer.
RMSE
Layer
Method
all
ReLU
reduction
MEM
out of memory
Convolution Layer 1 + 8
TreeHEM (T=2)
0.3391
0.2177 0.228
0.0948
TreeHEM (T=4)
0.3477
0.1011
MEM
out of memory
Convolution Layer 8 16
TreeHEM (T=2)
0.2116
0.1121
0.0688
TreeHEM (T=4)
0.1859
0.1032
0.065
MEM
out of memory
Convolution Layer 16 -
TreeHEM (T=2)
0.2085
0.1415
0.0334
TrecHEM (T=4)
0.236
0.1524
0.0468
MEM
out of memory
Convolution Layer 32 64
TreeHEM (T=2)
0.1204
0.0724
0.026
TreeHEM (T=4)
0.156
0.0764
0.045
Table 12: Comparing different input fitting methods in //128 8/64 16/32 with batch size 21 ModelNet10.
32/16 64/8 GMCN
Epochs / acc.
50
60
k-means
84,06
93,11
93,29
k-means+FM
82,36
92,72
92,81
random+FM
80,42
91,94
92,13
Eckart
80,48
90,82
91,04
Table 13: Different batch sizes and training epochs best-performing GMCN with four internal convolution layers (1/128 + 8/64 + 16/32 + 32/16 + 64/8 - 40) for ModelNet40.
Epochs acc.
batch size
#kernels
#params
training time (60)
87,35
87,62
5256
341641
2d 1 14h 57m
21
86,65 87,01
5256
341641
2d 12h 7m
Table 14: Results, parameters and training time for training and testing GMCN model with total of five convolution layers (1/128 + 8/64 - 16/32 - 32/16 - 64/8 128/4 + 40) on ModelNet40.
Epochs acc.
batch size
#kernels
#params
training time (60)
14
77,84
77.82
16008
1040521
4d 23h 17m
22

---

Published as a conference paper at ICLR 2022
In contrast to our results for ModelNet10, the achieved accuracy trails behind the best- pest-performing state-of- oo-tte--rttttt approaches when applied to ModelNet4O. It must be noted that these competitors include view-based methods and architectures that rely on purpose-buit built feature learning modules (e.g., relation-shape RS-CNN or piflecion-coovolutionccoooe results in fact competitive with other established convolution-ban based baselines, such VoxNet (83%), O-CNN (90%), and NormalNet Vanilla (87.7%).
in NormalNet RCC). Hence,
However, there are several options for further increasing the performance of GMCNs that were not explored in this work. For example, due to the long training times of our proof-of-concepl implementation. did not use data augmentation in the training process, making the model more susceptible to overfitting. Indeed, the training accuracy peaked at 95%, which is 8% higher than the test accuracy. Thus, non-negligible amount of overfitting in our best-performing model for ModelNet40 seems likely. Lastly, we expect that one important factor for maximizing GMCN performance the precision of the fitting of activation functions, as outlined below.
Similar to ModeINet10, the network's accuracy for ModelNet40 drops when increasing the number of convolution layers to a total five (see Table 14). Rather than overfitting, we ascribe this trend to the imprecise fitting of activation functions. This is supported by the results reported in Tables 10 and 11, which attribute the majority of the fitting error to ReLU fitting. Hence, we expect that the accuracy of GMCNs can be improved with more elaborate, high-quality fitting methods, which hope to explore in future work.
G ACTIVATION IN PARAMETER SPACE (APPLYING THE TRANSFER FUNCTION TO THE WEIGHTS)
At first glance, it seems possible that ReLU fitting in GMCNs could be simplified by applying the transfer function directly in parameter space, i.e., on the weight vector. In other words, performing an approximation of Gaussians by Diracs for the purpose of activation reduce the complexity of the approach.
Unfortunately, that not the case. Currently, negative Gaussian is on top of positive one, then the negative would reduce the contribution of the positive one. In extreme cases, the positive Gaussian could even disappear entirely. This would not be possible if the ReLu would be applied directly to the weight: All negative Gaussians would be cut away. while the positive ones would simply pass through. Thus, no gradient would ever reach a negative kernel Gaussian. Consequently, positive kernel Gaussians, in conjunction with the convolution and such a parameter space activation, would act as if there were no activation at all.
23


Reviews:
[{"review_id": "xTUxOSTnwCf", "reviewer": "Reviewer_Ao7K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: This paper provides a new solution to handle the high-dimensional problems.\n\nWeakness:\n\n1.   Compared with other methods, such as PointNet and PointNet++, the effectiveness and practicability of this method cannot be sufficiently demonstrated. It is recommended to validate GMCN on more complex datasets and to give a more adequate analysis. \n2.   TreeHEM is used to reduce the number of Gaussians in the mixture, and  $T=2$ works best in terms of computation time, memory consumption, and often fitting error. The determination of hyper-parameter or specific values are based on experience.\n3. The evaluation is quite insufficient as this paper claims that it is a general purpose deep learning The results are not very convincing on complex tasks.\n\nQuestions:\n1.   In Equation 4, there are $N\\times M$ terms, each term is $g_i(x)* g_j(x)$, so according to Equation 5, is the sum of each column substituted in the Relu function? In Equation 7, $B_i$ is unknown? $C$ in Equation 1 and $B$ in Equation 2 should be in bold form.\n2.   This paper mentions that \"The input GM is then processed by four consecutive blocks of Gaussian convolution  and ReLU fitting layers, where each doubles the number of feature channels and halves the number of Gaussians\". So is the first layer $4\\times 8$, and the  last layer $64\\times 128$, with $4$ Gaussians?", "summary_of_the_paper": "For the curse of dimensionality problem of CNNs, this paper presents the Gaussian mixture convolution network (GMCN), a method that alleviate the problem of high-dimension data. GMCN is a deep functional network method that represents data as functions. It fits a Gaussian mixture to result of transfer functions, such as RELUs. The effectiveness of this architecture is verified on MNIST and ModelNet10.", "summary_of_the_review": "The method presented in this paper is new, but more detailed analysis and evaluation are needed to make this work more solid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qBcys8ebwLS", "reviewer": "Reviewer_gTB4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n+ Novel, innovative approach\n+ Tackles an important problem\n\n## Weaknesses\n- Somewhat unclear how small the memory footprint is compared to other methods\n- Some central elements like dense fitting and reduction step not very well described\n- Somewhat unclear what the limiting factors are for scaling to more complex datasets\n\n\n## Detailed comments on weaknesses\n\n### Memory footprint\n\nA key argument for the method is that it allows working with sparse (e.g. volumetric) data without running into memory issues. However, after reading the paper I don't have a clear idea about the memory footprint. The number of features (8–>64) and the number of Gaussians (128–>8) doesn't seem very large, yet Table 3 reports ~6 GB memory (these numbers are not very useful, since we don't know the batch size). I wonder whether other methods like OctNets, PointNet etc. wouldn't perform better when matching the memory footprint instead of the number of parameters. After all, memory is the limiting factor when dealing with high-D data, not the number of parameters.\n\n\n### Central elements not described well\n\nWhile I understand the motivation and goals behind the dense fitting and reduction step, I could not follow the explanations in section 4.1. Since it's a fairly central part of the method, I think this section needs substantial improvement, both in terms of why things are done the way they are (e.g. Eq. 6+7) and how they are done exactly (TreeHEM).\n\n*[Update on this point after discussions] After some clarifications on the method, I am somewhat unsure about the role of potential negative weights in the convolution kernels. It appears that the method effectively does not use them (or, if so, in a very indirect way).*\n\n\n### What limits applicability to more complex datasets?\n\nIt seems like the methods doesn't work (yet) on ModelNet40. I wonder why this is the case. Is the number of Gaussians and/or feature maps that can be used too small? If that's the case, then does the argument about having a memory-efficient method really hold? If that's not the case, then what is the limiting factor?\n", "summary_of_the_paper": "The paper proposes a new convolution method to process sparse data in a memory-efficient manner. The authors represent input data, intermediate feature maps and convolution filters as mixtures of Gaussians, which allows computing the result of the convolution analytically. Since the number of mixture components would increase quickly from layer to layer and application of ReLU destroys the property of intermediate layers to be mixtures of Gaussians, they re-fit intermediate feature maps and develop a heuristic to reduce the number of mixture components. The authors evaluate their proposed method on MNIST (2d) and ModelNet10 (3D) and show competitive performance compared to classic PointNet/++ methods.", "summary_of_the_review": "I am a bit torn about the paper in its current form. On the one hand it proposes a new and potentially very useful approach. On the other hand, the execution leaves some things to be desired. This makes it a bit difficult to properly assess the benefits and limitations of the method compared to other approaches, in particular more recent ones than PointNet/++. If the authors can clarify the issues mentioned above and provide a convincing argument that the method might scale also to more complex use cases, I'm willing to increase my score.\n\n### Update after rebuttal/discussions\nI am still a bit torn, since the intuitions behind e.g. the dense fitting step are still not clear and the role of negative weights. Since some points were clarified, I would give the authors the benefit of the doubt, though.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "q0cL6kC-lDZ", "reviewer": "Reviewer_eQt8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " In a similar spirit to  Functional Data Analysis (Ramsay& Silverman) (e.g. with Functional PCA), the paper proposes to learn  functions with neural networks\nwhich is appropriate when data can be well embedded into GMs (e.g. 3d point cloud).\nThe paper proposes approaches for  GM fitting so that the output of the activation function (ReLu) becomes a GM.\n As the convolution of GMs produces a GMs with more components (Gaussians), reduction techniques are used to limit the number of components in the GMs as these pass through the network.\nThe resulting proposed approach shows good performance for classification.\n\n The cross product (convolution) of two GMMs has a closed form solution. (e.g. as already exploited  for   registration  https://doi.org/10.1109/TPAMI.2010.223 and in statistics http://www.jstor.org/stable/1271214 ). \nEquations  could be better presented e.g. in Eq (1) the weight $a$ should not be part of the Gaussian (as it integrates to  1) and Eq. 2 would best be written as a weighted sum. Results Eq (3) and (4) are known so section 3 acts as a reminder/ state of the art.  \nTraining time are said to not be competitive against state of the art.\n\nGiven that  GMs are chosen as functional representation,  why is ReLu chosen as activation? is there any other activation functions that would have transformed a GM into a GM (apart from the identity function) ? \nFor simplicity why is the activation function (ReLu) not applied to the means only (i.e. approximating Gaussian by Dirac ) while applying identity to covariances? (i.e. why is activation not applied in the parameter space of the Gaussian ?) \n", "summary_of_the_paper": "The paper proposes to rethink deep learning for learning functions \n(with functions being gaussian mixtures GMs, with  positive and/or negative weights unlike for PDFs) as opposed to tensors traditionally used for embedding data. \nFilters are learned as GMs since the cross product (convolution) of two GMMs has a closed form solution.\nApplying  an activation/transfer function (e.g. ReLu) to a GM does not produce a GM and \nthe paper proposes to approximate a GM from the output of the activation function. Such approach allows to define layers with input/output in GM forms.\n", "summary_of_the_review": "The paper is interesting and novel as it rethink neural networks  for learning functions. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8HTUpHlTn6l", "reviewer": "Reviewer_RmKw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n\nThis is an interesting work, where the main goal is to replace conventional/standard CNN operations by GM. And this, of course, can  open the door to other innovative approaches that substitute the conventional ones.\n\nWeaknesses\n\nAlthough I find the proposed approach interesting I would expect some more experimental evaluation to other related approaches and other datasets. Nevertheless, this does not substantially hamper the paper. Another issue that I was struggling with, and I think the main bottleneck of the paper, is the reduction step (namely the use of M Gaussians). And perhaps other choices for eq. (6) and (7) that are not discussed (is it possible to have alternatives ?). See my concern in the detailed review. \n\n", "summary_of_the_paper": "CNNs have made an undeniably successful/impact on computer vision tasks. Its core operation relies on grids of discrete data samples, where structure representation of data is used (e.g. vector matrices …). This paper tries to disrupt this well established idea used so far, by proposing a Gaussian of mixtures (GMs) for tackling the curse of dimensionality, that is not avoidable with the conventional CNN architectures. The resulting architecture is termed as a deep functional network.", "summary_of_the_review": "\nWhen describing the TreeHEM approach, for reducing/merging  (or killing some Gaussian components) I found it rather cumbersome as other methodologies can be used to prune the mixture components. Next, I leave several questions for the discussion:\n\n-it is known that the EM suffers from the problem of the initialization, that is, how many components (per node)  should we represent the data. This often leads to a model selection type of problem. So, how can we deal with this issue automatically in the proposed architecture ?\n\n-Also, it is mentioned that reduction of  the number to Gaussians must be performed. Here, it would be welcome to introduce some of the statistical metrics that are also available to perform this task. In this way it could be possible to have more than one approach, from which a comparison could be performed, and thus, providing more insight into how this could be properly tackled.\n\n-It is not clear the mapping between the proposed context and the clustering hierarchy in Vasconcelos & Lippman (1999) work (that is, levels, blocks etc…).\n\nIt is mentioned that a value of T = 2 Gaussian is the best choice. However, this is difficult to follow …\n\n-For completeness, please specify the meaning of the operation in (8).\n\n-When performing the pooling, the fitting process is also performed.Here, it is necessary to enlarge the receptive field. To accomplish this,  the reduction of the covariance diagonal must be decreased. The question is, how is it possible to find the best scaling factor for this purpose ? receptive field . Is it possible to act on the covariance uncertainty for this purpose ?\n\n-it is said that the k-means is used to estimate the initial center’s position. What happens if this procedure does not provide the best location for the centroids ?It is well known that this algorithm may have some limitations regarding this issue.\n\n-Since the proposal is based on the EM algorithm, a clear mapping with this should be done. Specifically, to describe explicitly the E-step and M-step\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a deep learning method that aims to address the curse-of-dimensionality problem of conventional convolutional neural networks (CNNs) by representing data and kernels with unconstrained ‘mixtures’ of Gaussians and exploiting the analytical form of the convolution of multidimensional Gaussian mixtures. Since the number of mixture components rapidly increases from layer to layer (after convolution) and common activation functions such as ReLU do not preserve the Gaussian Mixtures (GM), the paper proposes a fitting stage that fits a GM to the output of the transfer function and uses a heuristic to reduce the number of mixture components. Experiments are presented on MNIST (2d) and ModelNet10 (3D), which show competitive performance compared to other approaches such as classic CNNs, PointNet and PontNet++ methods.

There is somewhat an overall consensus on the novelty of the proposed approach and its potential to pave the way for further research. There were, however, several issues raised by the reviewers in terms of clarity, memory footprint and computational cost that limits the applicability of the method to more complex datasets. While the authors expanded on the dense fitting in their comments and in the revised version of the paper, it still remains unclear the role of the negative weights, as the dense fitting stage seems to constrain all the weights to be positive. In terms of memory footprint, the authors refer to the theoretical footprint and their implementation does not match this. Finally, it is acknowledged by the authors that the computational cost is a limitation that hinders the method from achieving competitive performance in more complex tasks.
Decision: Accept (Poster)