Neural Network Approximation based on Hausdorff distance of Zonotopes

In this work we theoretically contribute to neural network approximation by providing a novel tropical geometrical viewpoint to structured neural network compression . In particular , we show that the approximation error between two neural networks with ReLU activations and one hidden layer depends on the Hausdorff distance of the tropical zonotopes of the networks . This theorem comes as a first step towards a purely geometrical interpretation of neural network approximation . Based on this theoretical contribution , we propose geometrical methods that employ the K-means algorithm to compress the fully connected parts of ReLU activated deep neural networks . We analyze the error bounds of our algorithms theoretically based on our approximation theorem and evaluate them empirically on neural network compression . Our experiments follow a proof-of-concept strategy and indicate that our geometrical tools achieve improved performance over relevant tropical geometry techniques and can be competitive against non-tropical methods . 1 INTRODUCTION . Tropical geometry ( Maclagan & Sturmfels , 2015 ) is a mathematical field based on algebraic geometry and strongly linked to polyhedral and combinatorial geometry . It is built upon the tropical semiring which originally refers to the min-plus semiring ( Rmin , ∧ , + ) , but may also refer to the max-plus semiring ( Cuninghame-Green , 2012 ; Butkovič , 2010 ) . In our work , we follow the convention of the max-plus semiring ( Rmax , ∨ , + ) which replaces the classical operations of addition and multiplication by max and sum respectively . These operations turn polynomials into piecewise linear functions and , thus , having direct applications in neural networks . Tropical mathematics have a wide range of applications including game theory ( Akian et al. , 2012 ) , dynamical systems on weighted lattices ( Maragos , 2017 ) , finite state transducers ( Theodosis & Maragos , 2018 ; 2019 ) and convex regression ( Maragos & Theodosis , 2020 ; Tsilivis et al. , 2021 ) . Recently , there has been remarkable theoretical impact of tropical geometry in the study of neural networks and machine learning ( Maragos et al. , 2021 ) . Zhang et al . ( 2018 ) prove the equivalence of ReLU activated neural networks with tropical rational mappings . Furthermore , they show that zonotopes may serve as a computational mean for applications such as computing the bound shown in ( Montúfar et al. , 2014 ) regarding the number of the network ’ s linear regions . In a similar context , Charisopoulos & Maragos ( 2018 ) compute an upper bound to the number of linear regions of convolutional and MaxOut layers and propose a randomized algorithm for linear region counting . Moreover , Charisopoulos & Maragos ( 2017 ) study the morphological perceptron tropically and propose a training algorithm using the convex-concave procedure . Also , Dimitriadis & Maragos ( 2021 ) examine the training , pruning and shape properties of Morphological neural networks . Pruning ( Blalock et al. , 2020 ) or , generally , compressing neural networks gained interest in recent years due to the surprising capability of reducing the size of a neural network without compromising performance . As tropical geometry explains the mathematical structure of neural networks , pruning may be revisited tropically . Indeed , Alfarra et al . ( 2020 ) study the lottery ticket hypothesis ( Frankle & Carbin , 2018 ) from the tropical geometrical perspective and propose an unstructured compression algorithm based sparsifying the zonotope matrices of the network . Further , Smyrnis et al . ( 2020 ) propose a novel tropical division algorithm that applies to neural network minimization . A generalization of this method applies to multiclass networks ( Smyrnis & Maragos , 2020 ) . Contributions In our work , we contribute to structured neural network approximation from the mathematical viewpoint of tropical geometry : • We establish a novel bound on the approximation error between two neural networks with ReLU activations and one hidden layer . To prove this we bound the difference of the networks ’ tropical polynomials via the Hausdorff distance of their respective zonotopes . • Motivated by the geometrical bound , we derive two geometrical neural network compression methods that are based on zonotope reduction and employ K-means algorithm for clustering . Our algorithms are applied on the fully connected layers of ReLU activated neural networks . • Our geometrical algorithms are analyzed both theoretically and experimentally . The theoretical evaluation is based on the bounding theorem of neural network approximation error by the Hausdorff distance of the zonotopes . On the experimental part , we examine the performance of our algorithms on retaining the accuracy of convolutional neural networks when applying compression on their fully connected layers . Proofs , illustrations and experimental specifications are included in the supplementary material . 2 BACKGROUND ON TROPICAL GEOMETRY . We study tropical geometry from the viewpoint of the max-plus semiring ( Rmax , ∨ , + ) which is defined as the set Rmax = R ∪ { −∞ } equipped with two operations ( ∨ , + ) . Operation ∨ stands for max and + stands for sum . In max-plus algebra we define polynomials in the following way . Tropical polynomials A tropical polynomial f in d variables x = ( x1 , x2 , ... , xd ) T is defined as the function f ( x ) = max a∈A { aTx+ ca } ( 1 ) where A is a finite collection of vectors in Rd and ca is the corresponding monomial coefficient in Rmax = R ∪ { −∞ } . The set of such polynomials constitutes the semiring Rmax [ x ] of tropical polynomials . Note that each term aTx + ca corresponds to a hyperplane in Rd . We thus call the vectors a ∈ A the slopes of the tropical polynomial , and ca the respective bias . We allow slopes to be vectors with real coefficients rather than integer ones , as it is normally the case for regular polynomials . These polynomials are also referred to as signomials ( Duffin & Peterson , 1973 ) in the literature . Polytopes Polytopes have been studied extensively ( Ziegler , 2012 ; Grünbaum , 2013 ) and occur as a geometric tool for fields such as linear programming and optimization . They also have an important role in the analysis of neural networks . For instance , Zhang et al . ( 2018 ) ; Charisopoulos & Maragos ( 2018 ) show that linear regions of neural networks correspond to vertices of polytopes . Thus , the counting of linear regions reduces to a combinatorial geometry problem . In what follows , we explore this connection of tropical geometry with polytopes . Consider the tropical polynomial defined in ( 1 ) . The Newton Polytope associated to f ( x ) is defined as the convex hull of the slopes of the polynomial Newt ( f ) : = conv { a : a ∈ A } Furthermore , the Extended Newton Polytope of f ( x ) is defined as the convex hull of the slopes of the polynomial extended in the last dimension by the corresponding bias coefficient . ENewt ( f ) : = conv { ( aT , ca ) : a ∈ A } The following proposition computes the Extended Newton Polytope that occurs when a tropical operation is applied between two tropical polynomials . It will allow us to compute the polytope representation corresponding to a neural network ’ s hidden layer . Proposition 1 . ( Zhang et al. , 2018 ; Charisopoulos & Maragos , 2018 ) Let f , g ∈ Rmax [ x ] be two tropical polynomials . Then for the Extended Newton Polytopes it is true that ENewt ( f ∨ g ) = conv { ENewt ( f ) ∪ ENewt ( g ) } ENewt ( f + g ) = ENewt ( f ) ⊕ ENewt ( g ) Here ⊕ denotes Minkowski addition . In particular , for two sets A , B ⊆ Rd it is defined as A⊕B : = { a+ b |a ∈ A , b ∈ B } Corollary 1 . This result can be generalized to any finite set of polynomials using induction . Example 1 . Let f , g be two tropical polynomials in 2 variables , such that f ( x , y ) = max ( 2x+ y + 1 , 0 ) , g ( x , y ) = max ( x , y , 1 ) The tropical operations applied to these polynomials give f ∨ g = max ( 2x+ y + 1 , 0 , x , y , 1 ) f + g = max ( 3x+ y + 1 , x , 2x+ 2y + 1 , y , 2x+ y + 2 , 1 ) Fig . 1 illustrates the Extended Newton Polytopes of the original and the computed polynomials . The Extended Newton Polytope provides a geometrical representation of a tropical polynomial . In addition , it may be used to compute the values that the polynomial attains , as Proposition 2 indicates . Proposition 2 . ( Charisopoulos & Maragos , 2018 ) Let f ∈ Rmax [ x ] be a tropical polynomial in d variables . Let UF ( ENewt ( f ) ) be the points in the upper envelope of ENewt ( f ) , where upward direction is taken regarding the last dimension of Rd+1 . Then for each a ∈ A there exists a linear region of f on which f ( x ) = aTx+ ca if and only if ( aT , ca ) is a vertex of UF ( ENewt ( f ) ) . Example 2 . Using the polynomials from Example 1 we compute a reduced representation for f ∨ g. f ∨ g = max ( 2x+ y + 1 , 0 , x , y , 1 ) = max ( 2x+ y + 1 , x , y , 1 ) Indeed , the remaining terms correspond to the vertices of UF ( ENewt ( f ∨ g ) ) shown in Fig . 1 . 2.1 TROPICAL GEOMETRY OF NEURAL NETWORKS . Tropical geometry has the capability of expressing the mathematical structure of ReLU activated neural networks . We review some of the basic properties governing neural networks and introduce notation that will be used in our analysis . For this purpose , consider the ReLU activated neural network of Fig . 2 with one hidden layer . Network tropical equations The network consists of an input layer x = ( x1 , ... , xd ) , a hidden layer f = ( f1 , ... , fn ) with ReLU activations , an output layer v = ( v1 , ... , vm ) and two linear layers defined by the matrices A , C respectively . As illustrated in Fig . 2 we have Ai , : = ( aTi , bi ) for the first linear layer and Cj , : = ( cj1 , cj2 , ... , cjn ) for the second linear layer , as we ignore its biases . Furthermore , the output of the i−th component of the hidden layer f is computed as fi ( x ) = max ( d∑ k=1 aikxk + bi , 0 ) = max ( aTi x+ bi , 0 ) ( 2 ) We deduce that each fi is a tropical polynomial with two terms . It therefore follows that ENewt ( fi ) is a linear segment in Rd+1 . The components of the output layer may be computed as vj ( x ) = n∑ i=1 cjifi ( x ) = ∑ cji > 0 |cji|fi ( x ) − ∑ cji < 0 |cji|fi ( x ) = pj ( x ) − qj ( x ) ( 3 ) Tropical rational functions Functions pj , qj are both linear combinations of { fi } with positive coefficients , which implies that they are tropical polynomials . We conclude that every output node vi can be written as a difference of two tropical polynomials , which is defined as a tropical rational function . This indicates that the output layer of the neural network of Fig . 2 is equivalent to a tropical rational mapping . In fact , this result holds for deeper networks , in general , as demonstrated by Theorem 1 . Theorem 1 . ( Zhang et al. , 2018 ) Consider a ReLU activated deep neural network as a mapping F : Rd → Rm . Then F is equivalent to a tropical rational mapping . It is not known whether a tropical rational function r ( x ) admits an efficient geometric representation that determines its values { r ( x ) } for x ∈ Rd , as it holds for tropical polynomials with their polytopes in Proposition 2 . For this reason , we choose to work separately on the polytopes of the tropical polynomials pj , qj . Zonotopes Zonotopes are defined as the Minkowski sum of a finite set of line segments . They are a special case of polytopes that occur as a building block for our network . These geometrical structures provide a representation of the polynomials pj , qj in ( 3 ) that further allows us to build our compression algorithms . We use the notation Pj , Qj for the Extended Newton Polytopes of tropical polynomials pj , qj , respectively . Notice from ( 3 ) that for each component vj of the output pj , qj are written as linear combinations of tropical polynomials that correspond to linear segments . Thus Pj and Qj are zonotopes . We call Pj the positive zonotope , corresponding to the positive polynomial pj and Qj the negative one . Zonotope Generators Each neuron of the hidden layer represents geometrically a line segment contributing to the positive or negative zonotope . We thus call these line segments generators of the zonotope . The generators further receive the characterization positive or negative depending on the zonotope they contribute to . It is intuitive to expect that a zonotope gets more complex as its number of generators increases . In fact , each vertex of the zonotope can be computed as the sum of vertices of the generators , where we choose a vertex from each generating line segment , either 0 or cji ( aTi , bi ) . We summarize the above with the following extension of ( Charisopoulos & Maragos , 2018 ) . Proposition 3 . Pj , Qj are zonotopes in Rd+1 . For each vertex v of Pj there exists a subset of indices I+ of { 1 , 2 , ... , n } with cji > 0 , ∀i ∈ I+ such that v = ∑ i∈I+ cji ( aTi , bi ) . Similarly , a vertex u of Qj can be written as u = ∑ i∈I− cji ( aTi , bi ) where I− corresponds to cji < 0 , ∀i ∈ I− .

Summary: ***This is a educated guess review as the paper is outside my domain expertise.  This paper proposes a compression method using a framework based on geometrical zonotope reduction. The authors further analyze the error bounds of the proposed methods and compare its performance with modern pruning techniques. I think the main contribution of the work would be novelty of the proposed method.