Simplicial Complex Networks

1 INTRODUCTION . It is well-known that under mild assumptions on the activation function , a neural network with one hidden layer and a finite number of neurons can approximate continuous functions . This characteristic of neural networks is generally referred to as the universal approximation property . There are various theoretical universal approximators . For example , a result of the Stone-Weierstrass theorem Stone ( 1948 ) ; Cotter ( 1990 ) is that multivariate polynomials are dense in the space of continuous real valued functions defined over a hypercube . Another example is that the reproducing kernel Hilbert space ( RKHS ) associated with kernel functions with particular properties can be dense in the same space of functions . Kernel functions with this property are called universal kernels Micchelli et al . ( 2006 ) . A subsequent result of this theory is that the set of functions generated by a Gaussian process regression with an appropriate kernel can approximate any continuous function over a hypercube with arbitrary precision . Although multivariate polynomials and Gaussian processes also have this approximation property , each has practical limitations that cause neural networks to be used more often in practice compared to these approaches . For instance , polynomial interpolations may result a model that overfits to the data and suffers from a poor generalization , and Gaussian processes often become computationally intractable for a large number of training data Bernardo et al .. Neural networks , with an efficient structure for gradient computation using backpropagation , can be trained using gradient based optimization for large datasets in a tractable time . Moreover , in contrast to existing polynomial interpolations , neural networks generalize well in practice . Theoretical and empirical understanding of the generalization power of neural networks is an ongoing research Novak et al . ( 2018 ) ; Neyshabur et al . ( 2017 ) . Topological Data Analysis ( TDA ) , a geometric approach for data analysis , is a growing field which provides statistical and algorithmic methods to analyze the topological structures of data often referred to as point clouds . TDA methods mainly relied on deterministic methods until recently where statistical approaches were proposed for this purpose Carriere et al . ( 2017 ) ; Chazal & Michel ( 2017 ) . In general , TDA methods assume a point cloud in a metric space with an inducing distance ( e.g . Euclidean , Hausdorff , or Wasserstein distance ) between samples and build a topological structure upon point clouds . The topological structure is then used to extract geometric information from data Chazal & Michel ( 2017 ) . These models are not trained with gradient based approaches and they are generally limited to predetermined algorithms whose application to high dimensional spaces may be challenging Chazal ( 2016 ) . In this work , by leveraging geometrical perspective of TDA , we provide a class of restricted neural networks that preserve the universal approximation property and can be trained using a forward pass and the backpropagation algorithm . Motivated by the approximation theorem used to develop our method , Simplicial Complex Network ( SCN ) is chosen to refer these models . SCNs do not require an activation function and architecture search in the way that conventional neural networks do . Their hidden units are conceptually well defined , in contrast to feed-forward neural networks for which the role of a hidden unit is yet an ongoing problem . SCNs are discussed in more details in later sections . Our contribution can be summarized in building a novel class of neural networks which we believe can be used in the future for developing deep models that are interpretable , and robust to perturbations . The rest of this paper is organized as follows : Section 2 is specified for the explanation of SCNs and their training procedure . In section 3 , related works are explained . Sections 4 , 5 , and 6 are specified to experiments , limitations , and conclusion . 2 SIMPLICIAL COMPLEX NETWORKS . We first describe some necessary notation in the section 2.1 . In section 2.2 , we discuss the barycentric subdivision and the simplicial approximation theorem . In section 2.3 , we modify the barycentric subdivision in order to develop an approach that allows us to learn a subdivision of the input space into small simplexes . We then introduce a general framework for defining an SCN for regression tasks that simultaneously learns a subdivision of the input space and a piece-wise linear mapping where the linear pieces are defined on each simplex of the subdivision . 2.1 NOTATION . We consider a dataset D = { ( x ( i ) , y ( i ) ) } Ni=1 of N i.i.d . input/output pairs , where for each 1 ≤ i ≤ N , x ( i ) ∈ Rd and y ( i ) ∈ Rk . Let fθ denotes a mapping from the input to the output space in which θ represents its parameters . In a regression task , we wish to minimize J ( θ ) = E ( x ( i ) , y ( i ) ) ∼D [ L ( fθ ( x ( i ) ) , y ( i ) ) ] , where L is a given loss function . We indicate a d dimensional simplex ( d-simplex ) with vertices v0 , ... , vd with the notation [ v0 , v1 , ... , vd ] . For simplicity of the presentation , we further assume that each x ( i ) lies inside a d-simplex σ = [ v0 , v1 , ... , vd ] , where v0 is the origin , and v1 , ... vd are the standard basis vectors . Similarly , we assume each y ( i ) also lies in the standard probability k-simplex . 2.2 SIMPLICIAL APPROXIMATION . Simplicial approximation theorem allows the approximation of continuous functions using simplicial mappings . Before stating the theorem , we borrow a few definitions from the algebraic topology literature . Definition 1 . ( simplicial complex ) A simplicial complex K is a set of simplexes such that : 1 ) Every face of a simplex of K is in K. 2 ) The intersection of any two simplexes in K is a face of each of them . ( A simplicial complex can be informally defined as a set of simplexes glued together through their faces . ) Definition 2 . ( simplicial mapping ) A mapping between two simplicial complexes is called simplicial mapping if the images of the vertices are vertices . We also use the definition of a standard subdivisioning method which is used to break a d-simplex ( or any other simplicial complex ) into arbitrary small simplexes with the same dimension . Definition 3 . ( barycentric subdivision ) Barycentric subdivision ( BCS ) of a d-simplex K consists of ( d + 1 ) ! d-simplexes . Each d-simplex [ v0 , v1 , ... , vd ] out of these ( d + 1 ) ! simplexes is associated with a permutation p0 , p1 , ... , pd of the vertices of K such that vi denotes the barycenter ( centroid ) of p0 , p1 , ... , pi where 1 ≤ i ≤ n. Figure 2 ( a ) , ( b ) visualize examples of BCS for a 2-simplex , and a 3-simplex . Note that i-th BCS is the result of applying BCS to each simplex in the ( i− 1 ) -th BCS . Using these definitions , simplicial approximation theorem can be stated as follows , Theorem 1 . ( Simplicial Approximation Theorem ) Let X and Y be two simplicial complexes and f : X → Y be a continuous function . Then for an arbitrary , there exist sufficiently large k and l and a simplicial mapping g : X ( k ) → Y ( l ) approximating f such that supx∈X‖f ( x ) − g ( x ) ‖ < . X ( k ) and Y ( l ) represent the k-th and l-th barycentric subdivision of X and Y , respectively . In the appendix A , we provide a short topological proof for this theorem . Although the theorem provides a general approximation framework , its approximation is through using BCS of the input and output spaces . Each time BCS is applied , the number of simplexes is multiplied by ( d+ 1 ) ! and simplexes in higher order subdivisions become flatter and flatter Diaconis & Miclo ( 2011 ) . This fact limits the use of this subdivision algorithm in practice . Moreover , BCS subdivides input or output space completely independent of the data . In the next section , we modify the BCS to a data-driven approach which allows us to learn a suitable subdivision given data . Apart from BCS , in TDA , building simplicial complexes from data is often based on deterministic approaches . For instance , Vietoris-Rips complex Carlsson et al . ( 2006 ) ; Attali et al . ( 2013 ) is the set of all simplexes which their vertices are data points that their pair distances are less than a threshold . Čech complex Attali et al . ( 2013 ) ; Kerber & Sharathkumar ( 2013 ) , is the set of all those simplexes that completely lie in a closed ball with specific radius . While these non-parametric data driven methods can extract rich topological structures of the data using mathematical tools such as persistent homology , they are not often used as standard features in machine learning algorithms immediately Chazal & Michel ( 2017 ) ; Attali et al . ( 2013 ) . In the next section , we modify BCS to a parametric framework that allows automation of the subdivisioning process and it can be used directly during the training of an SCN . 2.3 AUTOMATIC SUBDIVISION . In the Algorithm 1 , we have shown the process of generating a random simplex from the set of all simplexes in BCS of a d-simplex . The algorithm gives an uncommon view for identification of a simplex in the BCS which is through a set of nested simplexes . We modify this view to a data driven approach in a way that BCS is a special case of the modified version . In the Algorithm 1 , for a random permutation P , N0 ⊃ N1 ⊃ ... ⊃ Nd form a nested set of simplexes . This fact is visualized in Figure 2 ( c ) , ( d ) . Nd indicates the simplex in the BCS that corresponds to P . Knowing the nested sequence uniquely determines a simplex in the BCS . Also , we note that each Ni can be obtained by replacing one of the vertices of Ni−1 with a new vertex inside or on the boundary ( closure ) of it . Thereby , since the new vertex is in the closure of Ni−1 , it can be represented as a convex combination of the vertices of Ni−1 . The weights of these convex combinations can be computed in a straightforward way as shown in the Algorithm 1 . Note that these weights are specified to the BCS . To build a data driven approach initially we eliminate two restrictions of the Algorithm 1 . First , we allow the number of repeats to be any arbitrary integer l rather fixing it to d. This number is later referred as the subdivision depth . Second , at each repeat , we let the weights to freely have any arbitrary values as long as they can be used as a convex combination weights . This removes the restriction of having deterministic weights specified for BSC . This modification allows us to learn the weights through an optimization process . A natural approach to make the subdivision process data-driven is to , instead of randomly sampling a simplex as in the Algorithm 1 , sample a simplex with the probability proportion to the likelihood that it contains data and update the subdivision parameters accordingly . However , the number of all possible simplexes grows exponentially as l increases , thereby making the computation of all these probabilities intractable . Alternatively , we sample from data and identify the simplex in the subdivision that contains the sample and use that simplex in an stochastic optimization framework to update the subdivision towards a desired one . But how can we identify the simplex that contains a given sample ? Our parameterization of the subdivisioning process using the nested sequence of simplexes along with the following lemma help to locate the input using a forward pass , Lemma 1 . Let x denote a sample inside a d-simplex σ = [ v0 , ... , vd ] . Assume x = ∑d i=0 wivi is represented as a convex combination of the vertices of σ . Let h = ∑d i=0 w ′ ivi be another point in the simplex written in the same format . Indicating the simplex with vertices of σ except vi replaced by h with σ−i , x lies in the simplex σ−j where j = arg mini wiw′i . Proof . x can be represented with the following convex combination which results the lemma , x = ( ∑ i 6=j ( wi − wj w′j w′i ) vi ) + wj w′j h Assume a sample x = ∑d i=0 wivi represented as a convex combination of vertices of a given d-simplex σ = [ v0 , v2 , ... , vd ] . Similar to the steps of Algorithm 1 and based on our view for identification of the target simplex through a nested set of simplexes , at the first step , a point is added inside the simplex using a convex combination of vertices of σ with weights w1 . Then lemma 1 can be used to locate the x in the σ and accordingly replace one of its vertices with the new vertex . This process is repeated with the new d-simplex up to l times to extract the simplex in the subdivision that contains x . Having the final simplex and a given cost function , parameters of the subdivision can be updated accordingly using the backpropagation and gradient descent framework . The procedure is formally shown in the Algorithm 2 . We refer the general procedure described in Algorithm 2 to as automatic subdivision . Note that the barycentric subdivision can be a particular outcome of the automatic subdivisioning . Algorithm 1 Generating a simplex in barycentric subdivision of a d-simplex input : d-simplex σ = [ v0 , v2 , ... , vd ] , permutation P = ( p0 , p1 , ... , pd ) initialize : N0 = σ , w = 1d , j = 0 repeat compute u = ∑d i=0 wi ( d+1 ) −jvi Set wpj = 0 Set Nj+1 = Nj with pj-th vertex replaced by u j = j + 1 until j = d return : Nd Algorithm 2 One gradient step in automatic subdivisioning of a d-simplex using one data sample input : d-simplex σ = [ v0 , v2 , ... , vd ] , sample x = ∑d i=0 wxivi , Θ = { w1 , ... , wl } , Loss L initialize : N0 = σ , j = 0 repeat Compute u = ∑d i=0 wjiNji Set k = argmini wxi wji Set Nj+1 = Nj with k-th vertex replaced with u Set wx as convex combination weights of x represented with vertices in Nj+1 j = j + 1 until j = l Θ = Θ− α∇ΘL ( Θ ) Project each wi ∈ Θ on the standard d-simplex

Summary: This paper presents a new neural network architecture called simplicial complex networks. The authors draw inspiration from topological data analysis in order to motivate this new architecture. The SCN architecture is motivated through a parametric sub-divisioning of the input space into a complex of d-dimensional simplices. The authors give a universal approximation theorem for a model of this type, and give a simple algorithm to perform the sub-divisioning. The resulting network output is a convex combination of vertices after l nested sub-divisions. The authors present a number of toy experiments to validate their claims.