Gaussian Mixture Convolution Networks

This paper proposes a novel method for deep learning based on the analytical convolution of multidimensional Gaussian mixtures . In contrast to tensors , these do not suffer from the curse of dimensionality and allow for a compact representation , as data is only stored where details exist . Convolution kernels and data are Gaussian mixtures with unconstrained weights , positions , and covariance matrices . Similar to discrete convolutional networks , each convolution step produces several feature channels , represented by independent Gaussian mixtures . Since traditional transfer functions like ReLUs do not produce Gaussian mixtures , we propose using a fitting of these functions instead . This fitting step also acts as a pooling layer if the number of Gaussian components is reduced appropriately . We demonstrate that networks based on this architecture reach competitive accuracy on Gaussian mixtures fitted to the MNIST and ModelNet data sets . 1 INTRODUCTION . Convolutional neural networks ( CNNs ) have led to the widespread adoption of deep learning in many disciplines . They operate on grids of discrete data samples , i.e. , vectors , matrices , and tensors , which are structured representations of data . While this is approach is sufficiently efficient in one and two dimensions , it suffers the curse of dimensionality : The amount of data is O ( dk ) , where d is the resolution of data , and k is the dimensionality . The exponential growth permeates all layers of a CNN , which must be scaled appropriately to learn a sufficient number of k-dimensional features . In three or more dimensions , the implied memory requirements thus quickly become intractable ( Maturana & Scherer , 2015 ; Zhirong Wu et al. , 2015 ) , leading researchers to propose specialized CNN architectures as a trade-off between mathematical exactness and performance ( Riegler et al. , 2017 ; Wang et al. , 2017 ) . In this work , we propose a novel deep learning architecture based on the analytical convolution of Gaussian mixtures ( GMs ) . While maintaining the elegance of conventional CNNs , our architecture does not suffer from the curse of dimensionality and is therefore well-suited for application to higher-dimensional problems . GMs are inherently unstructured and sparse , in the sense that no storage is required to represent empty data regions . In contrast to discretized k-dimensional volumes , this allows for a more compact representation of data across dimensions , preventing exponential memory requirements . In this regard , they are similar to point clouds ( Qi et al. , 2017a ) . However , GMs also encode the notion of a spatial extent . The fidelity of the representation directly depends on the number of Gaussians in the mixture , which means that , given an adequate fitting algorithm and sufficient resources , it can be arbitrarily adjusted . Hence , GMs allow trading representation accuracy for memory requirements on a more fine-grained level than voxel grids . In this work , we present our novel architecture , the Gaussian mixture convolution network ( GMCN ) , as an alternative to conventional CNNs . An important distinguishing feature of GMCNs is that both , data and learnable kernels , are represented using mixtures of Gaussian functions . We use Gaussians with unconstrained weights , positions , and covariances , enabling the mixtures to adapt more accurately to the shape of data than a grid of discrete samples could . Unlike probability models , our approach supports negative weights , which means that arbitrary data can be fitted . At their core , CNNs are deep neural networks , where each pixel or voxel is a neuron with distinct connections to other neurons , with shared weights and biases . On the other hand , GMCNs represent data as functions . Hence there are no distinct neurons , and the concept of connection does not exist . Our architecture could thus be more appropriately classified as a deep functional network . Like conventional ( discrete ) CNNs , our proposed GM learning approach supports several convolution layers , an increasing number of feature channels , and pooling , in order to learn and detect aspects of the input . For the convolution of two mixtures , a closed-form solution is used , which results in a new mixture . Conventional transfer functions like ReLUs do not produce a GM . However , a GM is required to feed the next convolution layer in a deep network . Therefore , we propose to fit a GM to the result of the transfer function . The fitting process can simultaneously act as a pooling layer if the number of fitted components is reduced appropriately in successive convolution layers . Unlike most point convolutional networks , which have per-point feature vectors , the feature channels of GMCNs are not restricted to share the same positions or covariance matrices . Compared to conventional CNNs , GMCNs exhibit a very compact theoretical memory footprint in 3 and more dimensions ( derivation and examples are included in Appendix A of this paper ) . With the presentation of GMCNs , we thus provide the following scientific contributions : • A deep learning architecture based on the analytical convolution of Gaussian mixtures . • A heuristic for fitting a Gaussian mixture to the result of a transfer function . • A fitting method to both reduce the number of Gaussians and act as a pooling layer . • A thorough evaluation of GMCNs , including ablation studies . 2 RELATED WORK . CNNs have been successfully applied to images ( e.g. , Krizhevsky et al . ( 2012 ) ) and small 3D voxel grids ( e.g. , Maturana & Scherer ( 2015 ) , and Ji et al . ( 2013 ) ) . However , it is not trivial to apply discrete convolution to large 3- or more dimensional problems due to the dimensional explosion . Riegler et al . ( 2017 ) reduce memory consumption by storing data in an octree , but the underlying problem remains . GMs have been used in 3D computer graphics , e.g. , by Jakob et al . ( 2011 ) for approximating volumetric data , and by Preiner et al . ( 2014 ) and Eckart et al . ( 2016a ) for a probabilistic model of point clouds . In particular , the work by Eckart et al . can be used to fit GMs to point clouds , which can then be used as input to our network ( see ablation study in Section 5.3 ) . We use the expectation-maximization ( EM ) algorithm proposed by Dempster et al . ( 1977 ) to fit input data , and EM equations for fitting small GMs to larger ones by Vasconcelos & Lippman ( 1999 ) . We refer to Ahmed et al . ( 2018 ) for a general overview of deep learning on 3D data . Like point clouds , GMs are unordered but with additional attributes ( weight and covariance ) . These additional attributes could be processed like feature vectors attached to points . Therefore , prior work on point clouds is relevant , whereby Guo et al . ( 2019 ) provide a good overview . Several authors propose point convolutional networks , e.g. , Li et al . ( 2018b ) , Xu et al . ( 2018 ) , Hermosilla et al . ( 2018 ) , Thomas et al . ( 2019 ) , Boulch ( 2019 ) , and Atzmon et al . ( 2018 ) . The one by Atzmon et al . is most similar to ours . They add a Gaussian with uniform variance for every point , creating a mixture that is a coarse fitting of the point cloud . After analytical convolution , the result is sampled at the original locations of the point cloud . However , information away from these points is lost . Effectively , such a restricted convolution creates a weighting function for neighboring points . Radial Basis Function Networks also learn a GM to detect patterns on the input data ( Broomhead & Lowe , 1988 ) . However , the goal of such networks differs from our approach . While their goal is , similar to MLPs , to detect patterns on a set of data points , our method uses radial basis functions to detect patterns using a convolution operator on another set of radial basis functions . 3 GAUSSIAN MIXTURES AND CONVOLUTION IN MULTIPLE DIMENSIONS . A multidimensional Gaussian function is defined as g ( x , b , C ) = 1√ ( 2π ) k det ( C ) e− 1 2 ( x−b ) TC−1 ( x−b ) , ( 1 ) where k is the number of dimensions , b the shift or center point , and C the shape ( covariance matrix ) of the Gaussian . A GM is a weighted sum of multiple such Gaussians : gm ( x , a , B , C ) = N∑ i=0 ai g ( x , Bi , Ci ) ( 2 ) A GM is not a probabilistic model since the weights ai can be negative and do not necessarily sum up to 1 . The convolution of two Gaussians according to Vinga ( 2004 ) is given by g ( x , b , C ) ∗ g ( x , β , Γ ) = g ( x , b+ β , C + Γ ) . ( 3 ) Due to the distributive property of convolution , we have : N∑ n=0 gn ( x ) ∗ M∑ m=0 gm ( x ) = N∑ n=0 M∑ m=0 gn ( x ) ∗ gm ( x ) , ( 4 ) where gi ( x ) is a short hand for ai g ( x , Bi , Ci ) . Clearly , the convolution of one mixture with N terms with another mixture with M terms yields a mixture with N ×M terms . 4 GAUSSIAN MIXTURE CONVOLUTION NETWORKS . Conventional CNNs typically consist of several feature channels , convolution layers , a transfer function , and pooling layers . In this section , we will show that all these components can be implemented by GMCNs as well . As a result , our overall architecture is similar to discrete CNNs , except that data and kernels are represented by Gaussian functions rather than pixels or voxels . Convolution layer . All batch elements and feature channels contain the same number of components , allowing for packing Gaussian data ( i.e. , weights , positions , and covariances ) into a single tensor . Like in conventional CNNs , each output channel ( Fo ) in a GMCN is the sum of all Fi input channels , convolved with individual kernels ( Figure 1 ) . In practice , this means that multiple convolution results are concatenated in the convolution layer . This results in a mixture with No = FiNiNk components , where Ni and Nk are component counts in input and kernel mixtures , respectively . Transfer function . The ReLU is a commonly used transfer function , whose ramp-like shape is defined by ϕ ( x ) = max ( 0 , x ) . When applying this transfer function to a mixture gm , we get ϕ ( gm ( x ) ) = max ( 0 , No∑ n=0 gn ( x ) ) . ( 5 ) We can use Equation 5 to evaluate the mixture at particular locations of x , as shown in Figure 2b . However , any subsequent convolution layers in a GMCN will again require a GM as input . Another problem inherent to the definition of GM convolution in Section 3 , is that the number of intermediate output components ( and thus implied memory requirements of hidden layers ) is set to increase with each convolution . We address both of these problems by introducing a dedicated fitting step that produces a new , smaller mixture to represent the result of ϕ ( gm ( x ) ) and act as input for subsequent GMCN layers .

Summary: The paper proposes to rethink deep learning for learning functions  (with functions being gaussian mixtures GMs, with  positive and/or negative weights unlike for PDFs) as opposed to tensors traditionally used for embedding data.  Filters are learned as GMs since the cross product (convolution) of two GMMs has a closed form solution. Applying  an activation/transfer function (e.g. ReLu) to a GM does not produce a GM and  the paper proposes to approximate a GM from the output of the activation function. Such approach allows to define layers with input/output in GM forms.