Gaussian Mixture Convolution Networks

This paper proposes a novel method for deep learning based on the analytical convolution of multidimensional Gaussian mixtures . In contrast to tensors , these do not suffer from the curse of dimensionality and allow for a compact representation , as data is only stored where details exist . Convolution kernels and data are Gaussian mixtures with unconstrained weights , positions , and covariance matrices . Similar to discrete convolutional networks , each convolution step produces several feature channels , represented by independent Gaussian mixtures . Since traditional transfer functions like ReLUs do not produce Gaussian mixtures , we propose using a fitting of these functions instead . This fitting step also acts as a pooling layer if the number of Gaussian components is reduced appropriately . We demonstrate that networks based on this architecture reach competitive accuracy on Gaussian mixtures fitted to the MNIST and ModelNet data sets . 1 INTRODUCTION . Convolutional neural networks ( CNNs ) have led to the widespread adoption of deep learning in many disciplines . They operate on grids of discrete data samples , i.e. , vectors , matrices , and tensors , which are structured representations of data . While this is approach is sufficiently efficient in one and two dimensions , it suffers the curse of dimensionality : The amount of data is O ( dk ) , where d is the resolution of data , and k is the dimensionality . The exponential growth permeates all layers of a CNN , which must be scaled appropriately to learn a sufficient number of k-dimensional features . In three or more dimensions , the implied memory requirements thus quickly become intractable ( Maturana & Scherer , 2015 ; Zhirong Wu et al. , 2015 ) , leading researchers to propose specialized CNN architectures as a trade-off between mathematical exactness and performance ( Riegler et al. , 2017 ; Wang et al. , 2017 ) . In this work , we propose a novel deep learning architecture based on the analytical convolution of Gaussian mixtures ( GMs ) . While maintaining the elegance of conventional CNNs , our architecture does not suffer from the curse of dimensionality and is therefore well-suited for application to higher-dimensional problems . GMs are inherently unstructured and sparse , in the sense that no storage is required to represent empty data regions . In contrast to discretized k-dimensional volumes , this allows for a more compact representation of data across dimensions , preventing exponential memory requirements . In this regard , they are similar to point clouds ( Qi et al. , 2017a ) . However , GMs also encode the notion of a spatial extent . The fidelity of the representation directly depends on the number of Gaussians in the mixture , which means that , given an adequate fitting algorithm and sufficient resources , it can be arbitrarily adjusted . Hence , GMs allow trading representation accuracy for memory requirements on a more fine-grained level than voxel grids . In this work , we present our novel architecture , the Gaussian mixture convolution network ( GMCN ) , as an alternative to conventional CNNs . An important distinguishing feature of GMCNs is that both , data and learnable kernels , are represented using mixtures of Gaussian functions . We use Gaussians with unconstrained weights , positions , and covariances , enabling the mixtures to adapt more accurately to the shape of data than a grid of discrete samples could . Unlike probability models , our approach supports negative weights , which means that arbitrary data can be fitted . At their core , CNNs are deep neural networks , where each pixel or voxel is a neuron with distinct connections to other neurons , with shared weights and biases . On the other hand , GMCNs represent data as functions . Hence there are no distinct neurons , and the concept of connection does not exist . Our architecture could thus be more appropriately classified as a deep functional network . Like conventional ( discrete ) CNNs , our proposed GM learning approach supports several convolution layers , an increasing number of feature channels , and pooling , in order to learn and detect aspects of the input . For the convolution of two mixtures , a closed-form solution is used , which results in a new mixture . Conventional transfer functions like ReLUs do not produce a GM . However , a GM is required to feed the next convolution layer in a deep network . Therefore , we propose to fit a GM to the result of the transfer function . The fitting process can simultaneously act as a pooling layer if the number of fitted components is reduced appropriately in successive convolution layers . Unlike most point convolutional networks , which have per-point feature vectors , the feature channels of GMCNs are not restricted to share the same positions or covariance matrices . Compared to conventional CNNs , GMCNs exhibit a very compact theoretical memory footprint in 3 and more dimensions ( derivation and examples are included in Appendix A of this paper ) . With the presentation of GMCNs , we thus provide the following scientific contributions : • A deep learning architecture based on the analytical convolution of Gaussian mixtures . • A heuristic for fitting a Gaussian mixture to the result of a transfer function . • A fitting method to both reduce the number of Gaussians and act as a pooling layer . • A thorough evaluation of GMCNs , including ablation studies . 2 RELATED WORK . CNNs have been successfully applied to images ( e.g. , Krizhevsky et al . ( 2012 ) ) and small 3D voxel grids ( e.g. , Maturana & Scherer ( 2015 ) , and Ji et al . ( 2013 ) ) . However , it is not trivial to apply discrete convolution to large 3- or more dimensional problems due to the dimensional explosion . Riegler et al . ( 2017 ) reduce memory consumption by storing data in an octree , but the underlying problem remains . GMs have been used in 3D computer graphics , e.g. , by Jakob et al . ( 2011 ) for approximating volumetric data , and by Preiner et al . ( 2014 ) and Eckart et al . ( 2016a ) for a probabilistic model of point clouds . In particular , the work by Eckart et al . can be used to fit GMs to point clouds , which can then be used as input to our network ( see ablation study in Section 5.3 ) . We use the expectation-maximization ( EM ) algorithm proposed by Dempster et al . ( 1977 ) to fit input data , and EM equations for fitting small GMs to larger ones by Vasconcelos & Lippman ( 1999 ) . We refer to Ahmed et al . ( 2018 ) for a general overview of deep learning on 3D data . Like point clouds , GMs are unordered but with additional attributes ( weight and covariance ) . These additional attributes could be processed like feature vectors attached to points . Therefore , prior work on point clouds is relevant , whereby Guo et al . ( 2019 ) provide a good overview . Several authors propose point convolutional networks , e.g. , Li et al . ( 2018b ) , Xu et al . ( 2018 ) , Hermosilla et al . ( 2018 ) , Thomas et al . ( 2019 ) , Boulch ( 2019 ) , and Atzmon et al . ( 2018 ) . The one by Atzmon et al . is most similar to ours . They add a Gaussian with uniform variance for every point , creating a mixture that is a coarse fitting of the point cloud . After analytical convolution , the result is sampled at the original locations of the point cloud . However , information away from these points is lost . Effectively , such a restricted convolution creates a weighting function for neighboring points . Radial Basis Function Networks also learn a GM to detect patterns on the input data ( Broomhead & Lowe , 1988 ) . However , the goal of such networks differs from our approach . While their goal is , similar to MLPs , to detect patterns on a set of data points , our method uses radial basis functions to detect patterns using a convolution operator on another set of radial basis functions . 3 GAUSSIAN MIXTURES AND CONVOLUTION IN MULTIPLE DIMENSIONS . A multidimensional Gaussian function is defined as g ( x , b , C ) = 1√ ( 2π ) k det ( C ) e− 1 2 ( x−b ) TC−1 ( x−b ) , ( 1 ) where k is the number of dimensions , b the shift or center point , and C the shape ( covariance matrix ) of the Gaussian . A GM is a weighted sum of multiple such Gaussians : gm ( x , a , B , C ) = N∑ i=0 ai g ( x , Bi , Ci ) ( 2 ) A GM is not a probabilistic model since the weights ai can be negative and do not necessarily sum up to 1 . The convolution of two Gaussians according to Vinga ( 2004 ) is given by g ( x , b , C ) ∗ g ( x , β , Γ ) = g ( x , b+ β , C + Γ ) . ( 3 ) Due to the distributive property of convolution , we have : N∑ n=0 gn ( x ) ∗ M∑ m=0 gm ( x ) = N∑ n=0 M∑ m=0 gn ( x ) ∗ gm ( x ) , ( 4 ) where gi ( x ) is a short hand for ai g ( x , Bi , Ci ) . Clearly , the convolution of one mixture with N terms with another mixture with M terms yields a mixture with N ×M terms . 4 GAUSSIAN MIXTURE CONVOLUTION NETWORKS . Conventional CNNs typically consist of several feature channels , convolution layers , a transfer function , and pooling layers . In this section , we will show that all these components can be implemented by GMCNs as well . As a result , our overall architecture is similar to discrete CNNs , except that data and kernels are represented by Gaussian functions rather than pixels or voxels . Convolution layer . All batch elements and feature channels contain the same number of components , allowing for packing Gaussian data ( i.e. , weights , positions , and covariances ) into a single tensor . Like in conventional CNNs , each output channel ( Fo ) in a GMCN is the sum of all Fi input channels , convolved with individual kernels ( Figure 1 ) . In practice , this means that multiple convolution results are concatenated in the convolution layer . This results in a mixture with No = FiNiNk components , where Ni and Nk are component counts in input and kernel mixtures , respectively . Transfer function . The ReLU is a commonly used transfer function , whose ramp-like shape is defined by ϕ ( x ) = max ( 0 , x ) . When applying this transfer function to a mixture gm , we get ϕ ( gm ( x ) ) = max ( 0 , No∑ n=0 gn ( x ) ) . ( 5 ) We can use Equation 5 to evaluate the mixture at particular locations of x , as shown in Figure 2b . However , any subsequent convolution layers in a GMCN will again require a GM as input . Another problem inherent to the definition of GM convolution in Section 3 , is that the number of intermediate output components ( and thus implied memory requirements of hidden layers ) is set to increase with each convolution . We address both of these problems by introducing a dedicated fitting step that produces a new , smaller mixture to represent the result of ϕ ( gm ( x ) ) and act as input for subsequent GMCN layers .

Summary: The paper proposes a new convolution method to process sparse data in a memory-efficient manner. The authors represent input data, intermediate feature maps and convolution filters as mixtures of Gaussians, which allows computing the result of the convolution analytically. Since the number of mixture components would increase quickly from layer to layer and application of ReLU destroys the property of intermediate layers to be mixtures of Gaussians, they re-fit intermediate feature maps and develop a heuristic to reduce the number of mixture components. The authors evaluate their proposed method on MNIST (2d) and ModelNet10 (3D) and show competitive performance compared to classic PointNet/++ methods.